MOA: Multi-Objective Alignment for Role-Playing Agents
Role-playing agents (RPAs) must simultaneously master many conflicting skills -- following multi-turn instructions, exhibiting domain knowledge, and adopting a consistent linguistic style. Existing work either relies on supervised fine-tuning (SFT) that over-fits surface cues and yields low diversity, or applies reinforcement learning (RL) that fails to learn multiple dimensions for comprehensive RPA optimization. We present MOA (Multi-Objective Alignment), a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs. MOA introduces a novel multi-objective optimization strategy that trains simultaneously on multiple fine-grained rubrics to boost optimization performance. Besides, to address the issues of model output diversity and quality, we have also employed thought-augmented rollout with off-policy guidance. Extensive experiments on challenging benchmarks such as PersonaGym and RoleMRC show that MOA enables an 8B model to match or even outperform strong baselines such as GPT-4o and Claude across numerous dimensions. This demonstrates the great potential of MOA in building RPAs that can simultaneously meet the demands of role knowledge, persona style, diverse scenarios, and complex multi-turn conversations.
693bc0719874a2a5e4ffb3dc
Chonghua Liao
admin_assigned
2025-12-12T13:08:36.298Z
667fdaee20ee9ac417c7708c
/avatars/69dfba6ff392643af1dcfe8af0a42ae9.svg
Chonghua Liao
ChonghuaLiao
user
ChonghuaLiao
693bc0719874a2a5e4ffb3dd
Ke Wang
693bc0719874a2a5e4ffb3de
Yuchuan Wu
admin_assigned
2025-12-12T13:08:15.799Z
677f6b5883442ad1180a0087
/avatars/fe928cbed47b21439db90e75264e9a72.svg
yuchuan wu
yuchuan123
user
yuchuan123
693bc0719874a2a5e4ffb3df
Fei Huang
693bc0719874a2a5e4ffb3e0
Yongbin Li
admin_assigned
2025-12-12T13:08:42.655Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
Yongbin-Li
user
Yongbin-Li
MOA: Multi-Objective Alignment for Role-Playing Agents
Role-playing agents (RPAs) must simultaneously master many conflicting skills -- following multi-turn instructions, exhibiting domain knowledge, and adopting a consistent linguistic style. Existing work either relies on supervised fine-tuning (SFT) that over-fits surface cues and yields low diversity, or applies reinforcement learning (RL) that fails to learn multiple dimensions for comprehensive RPA optimization. We present MOA (Multi-Objective Alignment), a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs. MOA introduces a novel multi-objective optimization strategy that trains simultaneously on multiple fine-grained rubrics to boost optimization performance. Besides, to address the issues of model output diversity and quality, we have also employed thought-augmented rollout with off-policy guidance. Extensive experiments on challenging benchmarks such as PersonaGym and RoleMRC show that MOA enables an 8B model to match or even outperform strong baselines such as GPT-4o and Claude across numerous dimensions. This demonstrates the great potential of MOA in building RPAs that can simultaneously meet the demands of role knowledge, persona style, diverse scenarios, and complex multi-turn conversations.
MOA, a reinforcement-learning framework, optimizes multiple dimensions of role-playing agents using multi-objective alignment and thought-augmented rollout, outperforming baselines in diverse scenarios and complex conversations.
reinforcement learning multi-objective optimization fine-grained rubrics thought-augmented rollout off-policy guidance PersonaGym RoleMRC