Manifold-Aware Exploration for Reinforcement Learning in Video Generation
Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.
69c22c68c105f45cb0481fe8
Mingzhe Zheng
claimed_verified
2026-03-24T09:21:18.284Z
62f0c4abe2999b231e5a893c
/avatars/90da268b877a7ffe6665075c84018a83.svg
Mingzhe Zheng
Dunge0nMaster
user
Dunge0nMaster
69c22c68c105f45cb0481fe9
Weijie Kong
69c22c68c105f45cb0481fea
Yue Wu
69c22c68c105f45cb0481feb
Dengyang Jiang
claimed_verified
2026-03-25T10:51:18.528Z
662a0f2d4bab737c1a279843
https://cdn-avatars.huggingface.co/v1/production/uploads/662a0f2d4bab737c1a279843/fC2p3mjMHkVpDQdEqkuR4.png
Dengyang Jiang
DyJiang
user
DyJiang
69c22c68c105f45cb0481fec
Yue Ma
69c22c68c105f45cb0481fed
Xuanhua He
69c22c68c105f45cb0481fee
Bin Lin
69c22c68c105f45cb0481fef
Kaixiong Gong
69c22c68c105f45cb0481ff0
Zhao Zhong
69c22c68c105f45cb0481ff1
Liefeng Bo
69c22c68c105f45cb0481ff2
Qifeng Chen
69c22c68c105f45cb0481ff3
Harry Yang
Manifold-Aware Exploration for Reinforcement Learning in Video Generation
Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.
Video generation methods face reliability challenges due to complex solution spaces and noise injection during ODE-to-SDE conversion, but a new approach constrains exploration within a learned data manifold to improve stability and quality.
Group Relative Policy Optimization video generation ODE-to-SDE conversion reward estimation manifold-aware SDE logarithmic curvature correction gradient norm equalizer trust region dual trust region periodic moving anchor stepwise constraints VideoAlign VQ MQ TA CLIPScore PickScore