GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via
  Regulated Clipping
Recently, GRPO-based reinforcement learning has shown remarkable progress in
optimizing flow-matching models, effectively improving their alignment with
task-specific rewards. Within these frameworks, the policy update relies on
importance-ratio clipping to constrain overconfident positive and negative
gradients. However, in practice, we observe a systematic shift in the
importance-ratio distribution-its mean falls below 1 and its variance differs
substantially across timesteps. This left-shifted and inconsistent distribution
prevents positive-advantage samples from entering the clipped region, causing
the mechanism to fail in constraining overconfident positive updates. As a
result, the policy model inevitably enters an implicit over-optimization
stage-while the proxy reward continues to increase, essential metrics such as
image quality and text-prompt alignment deteriorate sharply, ultimately making
the learned policy impractical for real-world use. To address this issue, we
introduce GRPO-Guard, a simple yet effective enhancement to existing GRPO
frameworks. Our method incorporates ratio normalization, which restores a
balanced and step-consistent importance ratio, ensuring that PPO clipping
properly constrains harmful updates across denoising timesteps. In addition, a
gradient reweighting strategy equalizes policy gradients over noise conditions,
preventing excessive updates from particular timestep regions. Together, these
designs act as a regulated clipping mechanism, stabilizing optimization and
substantially mitigating implicit over-optimization without relying on heavy KL
regularization. Extensive experiments on multiple diffusion backbones (e.g.,
SD3.5M, Flux.1-dev) and diverse proxy tasks demonstrate that GRPO-Guard
significantly reduces over-optimization while maintaining or even improving
generation quality.
69023886646208eac0d1f746
Jing Wang
69023886646208eac0d1f747
Jiajun Liang
69023886646208eac0d1f748
Jie Liu
69023886646208eac0d1f749
Henglin Liu
69023886646208eac0d1f74a
Gongye Liu
69023886646208eac0d1f74b
Jun Zheng
claimed_verified
2025-10-30T14:39:28.170Z
6381847a471a4550ff298c63
https://cdn-avatars.huggingface.co/v1/production/uploads/6381847a471a4550ff298c63/RTKepvX67R6pLiiUidpUO.png
Jun
zxbsmk
user
zxbsmk
69023886646208eac0d1f74c
Wanyuan Pang
69023886646208eac0d1f74d
Ao Ma
69023886646208eac0d1f74e
Zhenyu Xie
69023886646208eac0d1f74f
Xintao Wang
69023886646208eac0d1f750
Meng Wang
69023886646208eac0d1f751
Pengfei Wan
69023886646208eac0d1f752
Xiaodan Liang
GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via
  Regulated Clipping
Recently, GRPO-based reinforcement learning has shown remarkable progress in
optimizing flow-matching models, effectively improving their alignment with
task-specific rewards. Within these frameworks, the policy update relies on
importance-ratio clipping to constrain overconfident positive and negative
gradients. However, in practice, we observe a systematic shift in the
importance-ratio distribution-its mean falls below 1 and its variance differs
substantially across timesteps. This left-shifted and inconsistent distribution
prevents positive-advantage samples from entering the clipped region, causing
the mechanism to fail in constraining overconfident positive updates. As a
result, the policy model inevitably enters an implicit over-optimization
stage-while the proxy reward continues to increase, essential metrics such as
image quality and text-prompt alignment deteriorate sharply, ultimately making
the learned policy impractical for real-world use. To address this issue, we
introduce GRPO-Guard, a simple yet effective enhancement to existing GRPO
frameworks. Our method incorporates ratio normalization, which restores a
balanced and step-consistent importance ratio, ensuring that PPO clipping
properly constrains harmful updates across denoising timesteps. In addition, a
gradient reweighting strategy equalizes policy gradients over noise conditions,
preventing excessive updates from particular timestep regions. Together, these
designs act as a regulated clipping mechanism, stabilizing optimization and
substantially mitigating implicit over-optimization without relying on heavy KL
regularization. Extensive experiments on multiple diffusion backbones (e.g.,
SD3.5M, Flux.1-dev) and diverse proxy tasks demonstrate that GRPO-Guard
significantly reduces over-optimization while maintaining or even improving
generation quality.
GRPO-Guard enhances GRPO-based reinforcement learning by normalizing importance ratios and reweighting gradients, mitigating over-optimization in flow-matching models without heavy KL regularization.
GRPO reinforcement learning flow-matching models importance-ratio clipping policy update left-shifted distribution positive-advantage samples implicit over-optimization ratio normalization gradient reweighting diffusion backbones SD3.5M Flux.1-dev proxy tasks generation quality