Unified Personalized Reward Model for Vision Generation
Recent advancements in multimodal reward models (RMs) have significantly propelled the development of visual generation. Existing frameworks typically adopt Bradley-Terry-style preference modeling or leverage generative VLMs as judges, and subsequently optimize visual generation models via reinforcement learning. However, current RMs suffer from inherent limitations: they often follow a one-size-fits-all paradigm that assumes a monolithic preference distribution or relies on fixed evaluation rubrics. As a result, they are insensitive to content-specific visual cues, leading to systematic misalignment with subjective and context-dependent human preferences. To this end, inspired by human assessment, we propose UnifiedReward-Flex, a unified personalized reward model for vision generation that couples reward modeling with flexible and context-adaptive reasoning. Specifically, given a prompt and the generated visual content, it first interprets the semantic intent and grounds on visual evidence, then dynamically constructs a hierarchical assessment by instantiating fine-grained criteria under both predefined and self-generated high-level dimensions. Our training pipeline follows a two-stage process: (1) we first distill structured, high-quality reasoning traces from advanced closed-source VLMs to bootstrap SFT, equipping the model with flexible and context-adaptive reasoning behaviors; (2) we then perform direct preference optimization (DPO) on carefully curated preference pairs to further strengthen reasoning fidelity and discriminative alignment. To validate the effectiveness, we integrate UnifiedReward-Flex into the GRPO framework for image and video synthesis, and extensive results demonstrate its superiority.
6981774fce18b186280960be
Yibin Wang
claimed_verified
2026-02-04T12:33:05.113Z
654c6845bac6e6e49895a5b5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/KXQaAxulqr8jNBSpEaYM4.png
SII-Yibin Wang
CodeGoat24
user
CodeGoat24
6981774fce18b186280960bf
Yuhang Zang
claimed_verified
2026-02-06T18:57:12.948Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
yuhangzang
user
yuhangzang
6981774fce18b186280960c0
Feng Han
6981774fce18b186280960c1
Jiazi Bu
claimed_verified
2026-03-23T11:03:41.248Z
6564a63de8d6a163c0a4fd2b
/avatars/a4b7cc9ea6e4023f150fd745cab3a883.svg
Azusa
Bujiazi
user
Bujiazi
6981774fce18b186280960c2
Yujie Zhou
claimed_verified
2026-03-23T11:01:38.529Z
64b51fd8bcfd8542d6473d9a
/avatars/ceaa73b79f448996187f07733d96b800.svg
yujie
yujieouo
user
yujieouo
6981774fce18b186280960c3
Cheng Jin
6981774fce18b186280960c4
Jiaqi Wang
claimed_verified
2026-02-17T15:50:10.260Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
myownskyW7
user
myownskyW7
Unified Personalized Reward Model for Vision Generation
Recent advancements in multimodal reward models (RMs) have significantly propelled the development of visual generation. Existing frameworks typically adopt Bradley-Terry-style preference modeling or leverage generative VLMs as judges, and subsequently optimize visual generation models via reinforcement learning. However, current RMs suffer from inherent limitations: they often follow a one-size-fits-all paradigm that assumes a monolithic preference distribution or relies on fixed evaluation rubrics. As a result, they are insensitive to content-specific visual cues, leading to systematic misalignment with subjective and context-dependent human preferences. To this end, inspired by human assessment, we propose UnifiedReward-Flex, a unified personalized reward model for vision generation that couples reward modeling with flexible and context-adaptive reasoning. Specifically, given a prompt and the generated visual content, it first interprets the semantic intent and grounds on visual evidence, then dynamically constructs a hierarchical assessment by instantiating fine-grained criteria under both predefined and self-generated high-level dimensions. Our training pipeline follows a two-stage process: (1) we first distill structured, high-quality reasoning traces from advanced closed-source VLMs to bootstrap SFT, equipping the model with flexible and context-adaptive reasoning behaviors; (2) we then perform direct preference optimization (DPO) on carefully curated preference pairs to further strengthen reasoning fidelity and discriminative alignment. To validate the effectiveness, we integrate UnifiedReward-Flex into the GRPO framework for image and video synthesis, and extensive results demonstrate its superiority.
UnifiedReward-Flex combines reward modeling with flexible, context-adaptive reasoning to improve visual generation by dynamically constructing hierarchical assessments based on semantic intent and visual evidence.
multimodal reward models visual generation Bradley-Terry-style preference modeling generative VLMs reinforcement learning preference optimization direct preference optimization structured reasoning traces semantic intent visual evidence hierarchical assessment flexible reasoning context-adaptive reasoning GRPO framework image synthesis video synthesis