One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-specific preferences and automatically provide individualized feedback. However, developing these models faces two critical challenges: the scarcity of feedback from individual users and the need for efficient adaptation to unseen users. We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation. To realize this, we propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem. Specifically, we represent each user's reward model as a weighted combination of base reward functions, and optimize the initialization of these weights using a Model-Agnostic Meta-Learning (MAML)-style framework to support fast adaptation under limited feedback. To ensure robustness, we introduce the Robust Personalization Objective (RPO), which places greater emphasis on hard-to-learn users during meta optimization. Extensive experiments on personalized preference datasets validate that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines.
69782efc026bdf0473116e26
Hongru Cai
claimed_verified
2026-01-27T09:42:36.597Z
653a111eee5888edef9182cf
https://cdn-avatars.huggingface.co/v1/production/uploads/653a111eee5888edef9182cf/7jQ08JDk2UEBla91At8QR.jpeg
Hongru Cai
HongruCai
user
HongruCai
69782efc026bdf0473116e27
Yongqi Li
69782efc026bdf0473116e28
Tiezheng Yu
69782efc026bdf0473116e29
Fengbin Zhu
69782efc026bdf0473116e2a
Wenjie Wang
69782efc026bdf0473116e2b
Fuli Feng
69782efc026bdf0473116e2c
Wenjie Li
One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-specific preferences and automatically provide individualized feedback. However, developing these models faces two critical challenges: the scarcity of feedback from individual users and the need for efficient adaptation to unseen users. We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation. To realize this, we propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem. Specifically, we represent each user's reward model as a weighted combination of base reward functions, and optimize the initialization of these weights using a Model-Agnostic Meta-Learning (MAML)-style framework to support fast adaptation under limited feedback. To ensure robustness, we introduce the Robust Personalization Objective (RPO), which places greater emphasis on hard-to-learn users during meta optimization. Extensive experiments on personalized preference datasets validate that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines.
Meta Reward Modeling reformulates personalized reward modeling as a meta-learning problem to enable efficient adaptation to individual users with limited feedback.
meta-learning reward modeling personalized reward models Model-Agnostic Meta-Learning MAML few-shot personalization Robust Personalization Objective RPO