FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable
  Reasoning
Reinforcement learning with verifiable rewards (RLVR) has emerged as a
promising paradigm for enhancing the reasoning capabilities of large language
models (LLMs). In this context, models explore reasoning trajectories and
exploit rollouts with correct answers as positive signals for policy
optimization. However, these rollouts might involve flawed patterns such as
answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are
rewarded identically to fully correct ones, causing policy models to
internalize these unreliable reasoning patterns. In this work, we first conduct
a systematic study of flawed-positive rollouts in RL and find that they enable
rapid capability gains during the early optimization stage, while constraining
reasoning capability later by reinforcing unreliable patterns. Building on
these insights, we propose Flawed-Aware Policy Optimization (FAPO), which
presents a parameter-free reward penalty for flawed-positive rollouts, enabling
the policy to leverage them as useful shortcuts in the warm-up stage, securing
stable early gains, while gradually shifting optimization toward reliable
reasoning in the later refinement stage. To accurately and comprehensively
detect flawed-positive rollouts, we introduce a generative reward model (GenRM)
with a process-level reward that precisely localizes reasoning errors.
Experiments show that FAPO is effective in broad domains, improving outcome
correctness, process reliability, and training stability without increasing the
token budget.
69003b3d22d452aac6dd43b0
Yuyang Ding
claimed_verified
2025-10-30T14:43:43.261Z
626cf0f65651e31a7a2b9779
https://cdn-avatars.huggingface.co/v1/production/uploads/626cf0f65651e31a7a2b9779/xES0qcewzvfSIy_0Sqd0u.png
Ding
dyyyyyyyy
user
dyyyyyyyy
69003b3d22d452aac6dd43b1
Chi Zhang
69003b3d22d452aac6dd43b2
Juntao Li
69003b3d22d452aac6dd43b3
Haibin Lin
69003b3d22d452aac6dd43b4
Xin Liu
69003b3d22d452aac6dd43b5
Min Zhang
FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable
  Reasoning
Reinforcement learning with verifiable rewards (RLVR) has emerged as a
promising paradigm for enhancing the reasoning capabilities of large language
models (LLMs). In this context, models explore reasoning trajectories and
exploit rollouts with correct answers as positive signals for policy
optimization. However, these rollouts might involve flawed patterns such as
answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are
rewarded identically to fully correct ones, causing policy models to
internalize these unreliable reasoning patterns. In this work, we first conduct
a systematic study of flawed-positive rollouts in RL and find that they enable
rapid capability gains during the early optimization stage, while constraining
reasoning capability later by reinforcing unreliable patterns. Building on
these insights, we propose Flawed-Aware Policy Optimization (FAPO), which
presents a parameter-free reward penalty for flawed-positive rollouts, enabling
the policy to leverage them as useful shortcuts in the warm-up stage, securing
stable early gains, while gradually shifting optimization toward reliable
reasoning in the later refinement stage. To accurately and comprehensively
detect flawed-positive rollouts, we introduce a generative reward model (GenRM)
with a process-level reward that precisely localizes reasoning errors.
Experiments show that FAPO is effective in broad domains, improving outcome
correctness, process reliability, and training stability without increasing the
token budget.
Flawed-Aware Policy Optimization (FAPO) enhances reinforcement learning with verifiable rewards by penalizing flawed-positive rollouts, improving reasoning capability and training stability in large language models.
reinforcement learning verifiable rewards large language models reasoning trajectories policy optimization flawed-positive rollouts parameter-free reward penalty generative reward model process-level reward reasoning errors