RLoop: An Self-Improving Framework for Reinforcement Learning with
  Iterative Policy Initialization
While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for
training large reasoning models, its training dynamics harbor a critical
challenge: RL overfitting, where models gain training rewards but lose
generalization. Our analysis reveals this is driven by policy
over-specialization and catastrophic forgetting of diverse solutions generated
during training. Standard optimization discards this valuable inter-step policy
diversity. To address this, we introduce RLoop, a self-improving framework
built on iterative policy initialization. RLoop transforms the standard
training process into a virtuous cycle: it first uses RL to explore the
solution space from a given policy, then filters the successful trajectories to
create an expert dataset. This dataset is used via Rejection-sampling
Fine-Tuning (RFT) to refine the initial policy, creating a superior starting
point for the next iteration. This loop of exploration and exploitation via
iterative re-initialization effectively converts transient policy variations
into robust performance gains. Our experiments show RLoop mitigates forgetting
and substantially improves generalization, boosting average accuracy by 9% and
pass@32 by over 15% compared to vanilla RL.
6912a610a644ba07c499c6ed
Zeng Zhiyuan
6912a610a644ba07c499c6ee
Jiashuo Liu
6912a610a644ba07c499c6ef
Zhangyue Yin
6912a610a644ba07c499c6f0
Ge Zhang
claimed_verified
2025-11-18T14:09:42.042Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
zhangysk
user
zhangysk
6912a610a644ba07c499c6f1
Wenhao Huang
6912a610a644ba07c499c6f2
Xipeng Qiu
RLoop: An Self-Improving Framework for Reinforcement Learning with
  Iterative Policy Initialization
While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for
training large reasoning models, its training dynamics harbor a critical
challenge: RL overfitting, where models gain training rewards but lose
generalization. Our analysis reveals this is driven by policy
over-specialization and catastrophic forgetting of diverse solutions generated
during training. Standard optimization discards this valuable inter-step policy
diversity. To address this, we introduce RLoop, a self-improving framework
built on iterative policy initialization. RLoop transforms the standard
training process into a virtuous cycle: it first uses RL to explore the
solution space from a given policy, then filters the successful trajectories to
create an expert dataset. This dataset is used via Rejection-sampling
Fine-Tuning (RFT) to refine the initial policy, creating a superior starting
point for the next iteration. This loop of exploration and exploitation via
iterative re-initialization effectively converts transient policy variations
into robust performance gains. Our experiments show RLoop mitigates forgetting
and substantially improves generalization, boosting average accuracy by 9% and
pass@32 by over 15% compared to vanilla RL.
RLoop, a self-improving framework using iterative policy initialization and Rejection-sampling Fine-Tuning, mitigates overfitting and enhances generalization in Reinforcement Learning for Verifiable Rewards.
Reinforcement Learning for Verifiable Rewards RLVR RL overfitting policy over-specialization catastrophic forgetting iterative policy initialization RLoop Rejection-sampling Fine-Tuning RFT exploration exploitation transient policy variations robust performance gains pass@32