RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.
69dde1f8aea31e227c22c670
Haozhe Wang
claimed_verified
2026-04-16T20:22:16.489Z
65bf52f0259bc6caeb74f8bf
/avatars/b38392e954466df784a5760ded5df804.svg
Haozhe Wang
JasperHaozhe
user
JasperHaozhe
69dde1f8aea31e227c22c671
Cong Wei
claimed_verified
2026-04-16T20:22:14.439Z
64f8e358766ff9f3d2b0de84
https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
Cong Wei
CongWei1230
user
CongWei1230
69dde1f8aea31e227c22c672
Weiming Ren
69dde1f8aea31e227c22c673
Jiaming Liu
claimed_verified
2026-04-16T08:08:31.955Z
637c941588699fba70e29f70
https://cdn-avatars.huggingface.co/v1/production/uploads/637c941588699fba70e29f70/b6G_QZkT-MhE47dx87i0d.png
LIU JIAMING
jamesliu1217
user
jamesliu1217
69dde1f8aea31e227c22c674
Fangzhen Lin
69dde1f8aea31e227c22c675
Wenhu Chen
RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.
Training reward models to generate multi-dimensional critiques improves visual generation through both enhanced reinforcement learning rewards and test-time refinement loops, achieving state-of-the-art performance with reduced training data requirements.
reward models visual generation preference prediction reinforcement learning Generate-Critique-Refine loop Preference-Anchored Rationalization PARROT rationalization structured reasoning text-to-image generation image-editing generation RL-based fine-tuning