P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling
Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.
698ec6ebcace060ff123aee2
Pinyi Zhang
698ec6ebcace060ff123aee3
Ting-En Lin
claimed_verified
2026-04-15T12:09:51.952Z
62e0ef42edb0462c8d51818d
https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg
Ting-En Lin
tnlin
user
tnlin
698ec6ebcace060ff123aee4
Yuchuan Wu
698ec6ebcace060ff123aee5
Jingyang Chen
698ec6ebcace060ff123aee6
Zongqi Wang
698ec6ebcace060ff123aee7
Hua Yang
698ec6ebcace060ff123aee8
Ze Xu
698ec6ebcace060ff123aee9
Fei Huang
698ec6ebcace060ff123aeea
Kai Zhang
698ec6ebcace060ff123aeeb
Yongbin Li
P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling
Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.
Personalized generative reward models address challenges in adapting language model responses to individual user preferences by using structured evaluation chains and dual-granularity scaling mechanisms for improved generalization and accuracy.
personalized alignment large language models reinforcement learning reward models generative reward models test-time user-based scaling adaptive personas scoring rubrics user clustering user prototypes dual-granularity scaling preference signals generalization out-of-distribution dataset