Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning
Recent studies have demonstrated significant progress in aligning text-to-image diffusion models with human preference via Reinforcement Learning from Human Feedback. However, while existing methods achieve high scores on automated reward metrics, they often lead to Preference Mode Collapse (PMC)-a specific form of reward hacking where models converge on narrow, high-scoring outputs (e.g., images with monolithic styles or pervasive overexposure), severely degrading generative diversity. In this work, we introduce and quantify this phenomenon, proposing DivGenBench, a novel benchmark designed to measure the extent of PMC. We posit that this collapse is driven by over-optimization along the reward model's inherent biases. Building on this analysis, we propose Directional Decoupling Alignment (D^2-Align), a novel framework that mitigates PMC by directionally correcting the reward signal. Specifically, our method first learns a directional correction within the reward model's embedding space while keeping the model frozen. This correction is then applied to the reward signal during the optimization process, preventing the model from collapsing into specific modes and thereby maintaining diversity. Our comprehensive evaluation, combining qualitative analysis with quantitative metrics for both quality and diversity, reveals that D^2-Align achieves superior alignment with human preference.
695c6b646aa73bc11f0913af
Chubin Chen
695c6b646aa73bc11f0913b0
Sujie Hu
695c6b646aa73bc11f0913b1
Jiashu Zhu
695c6b646aa73bc11f0913b2
Meiqi Wu
695c6b646aa73bc11f0913b3
Jintao Chen
695c6b646aa73bc11f0913b4
Yanxun Li
695c6b646aa73bc11f0913b5
Nisha Huang
695c6b646aa73bc11f0913b6
Chengyu Fang
695c6b646aa73bc11f0913b7
Jiahong Wu
695c6b646aa73bc11f0913b8
Xiangxiang Chu
695c6b646aa73bc11f0913b9
Xiu Li
Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning
Recent studies have demonstrated significant progress in aligning text-to-image diffusion models with human preference via Reinforcement Learning from Human Feedback. However, while existing methods achieve high scores on automated reward metrics, they often lead to Preference Mode Collapse (PMC)-a specific form of reward hacking where models converge on narrow, high-scoring outputs (e.g., images with monolithic styles or pervasive overexposure), severely degrading generative diversity. In this work, we introduce and quantify this phenomenon, proposing DivGenBench, a novel benchmark designed to measure the extent of PMC. We posit that this collapse is driven by over-optimization along the reward model's inherent biases. Building on this analysis, we propose Directional Decoupling Alignment (D^2-Align), a novel framework that mitigates PMC by directionally correcting the reward signal. Specifically, our method first learns a directional correction within the reward model's embedding space while keeping the model frozen. This correction is then applied to the reward signal during the optimization process, preventing the model from collapsing into specific modes and thereby maintaining diversity. Our comprehensive evaluation, combining qualitative analysis with quantitative metrics for both quality and diversity, reveals that D^2-Align achieves superior alignment with human preference.
Researchers identify and address Preference Mode Collapse in text-to-image diffusion models through a novel framework that maintains generative diversity while improving human preference alignment.
text-to-image diffusion models Reinforcement Learning from Human Feedback Preference Mode Collapse reward hacking DivGenBench Directional Decoupling Alignment D$^2$-Align reward model embedding space reward signal optimization process