VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation
Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models.
695c7d256aa73bc11f091459
Shikun Sun
695c7d256aa73bc11f09145a
Liao Qu
claimed_verified
2026-01-06T09:57:53.171Z
64b796079ebb7e6c7ddcdabf
/avatars/51af43cab078705b8745b4f942f542e5.svg
Liao Qu
leo1117
user
leo1117
695c7d256aa73bc11f09145b
Huichao Zhang
695c7d256aa73bc11f09145c
Yiheng Liu
695c7d256aa73bc11f09145d
Yangyang Song
695c7d256aa73bc11f09145e
Xian Li
695c7d256aa73bc11f09145f
Xu Wang
695c7d256aa73bc11f091460
Yi Jiang
695c7d256aa73bc11f091461
Daniel K. Du
695c7d256aa73bc11f091462
Xinglong Wu
695c7d256aa73bc11f091463
Jia Jia
VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation
Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models.
Visual autoregressive models face training instability due to asynchronous policy conflicts, which are addressed through a novel framework enhancing group relative policy optimization with intermediate rewards, dynamic time-step reweighting, and mask propagation algorithms.
AutoRegressive diffusion Visual AutoRegressive reinforcement learning Group Relative Policy Optimization intermediate reward dynamic time-step reweighting mask propagation Reward Feedback Learning visual generation