Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.
69b37ad72ad6c5d4cf486a95
Xiangyu Zhao
69b37ad72ad6c5d4cf486a96
Peiyuan Zhang
admin_assigned
2026-03-13T08:52:10.901Z
6710be3e6d1b33cf24417e38
/avatars/f60bc9a67bb58f5997cbcc28cb93c079.svg
zpy
zpy777
user
zpy777
69b37ad72ad6c5d4cf486a97
Junming Lin
claimed_verified
2026-03-13T08:12:48.248Z
65f7a2a72f5698447c774e17
/avatars/e1a82df58af02c0ff08287e78f438480.svg
Junming Lin
mjuicem
user
mjuicem
69b37ad72ad6c5d4cf486a98
Tianhao Liang
69b37ad72ad6c5d4cf486a99
Yuchen Duan
69b37ad72ad6c5d4cf486a9a
Shengyuan Ding
admin_assigned
2026-03-13T08:54:05.785Z
646cd947da8e99940b6e55cf
https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
Shengyuan Ding
ChrisDing1105
user
ChrisDing1105
69b37ad72ad6c5d4cf486a9b
Changyao Tian
admin_assigned
2026-03-13T08:53:59.924Z
64b7475efa7eabaae5f7ba94
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7475efa7eabaae5f7ba94/YLv44PZM6tw1sxACY1-U_.png
Changyao Tian
Changyao
user
Changyao
69b37ad72ad6c5d4cf486a9c
Yuhang Zang
claimed_verified
2026-03-13T08:12:50.179Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
yuhangzang
user
yuhangzang
69b37ad72ad6c5d4cf486a9d
Junchi Yan
admin_assigned
2026-03-13T08:53:53.408Z
667289f903c802764985d8c6
/avatars/916befcbf0e52ce56be49617f31c7bb2.svg
Junchi Yan
Rethinker
user
Rethinker
69b37ad72ad6c5d4cf486a9e
Xue Yang
Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.
Reinforcement learning framework with novel reward modeling and benchmarking approaches improves fidelity and instruction adherence in image editing and text-to-image generation.
reinforcement learning reward models image editing text-to-image generation hallucinations reward modeling data curation FIRM-Bench Base-and-Bonus reward strategy Consistency-Modulated Execution Quality-Modulated Alignment