From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space
While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model's existing output distribution. Optimizing the marginal distribution P(y) in the Pre-train Space addresses this bottleneck by encoding reasoning ability and preserving broad exploration capacity. Yet, conventional pre-training relies on static corpora for passive learning, leading to a distribution shift that hinders targeted reasoning enhancement. In this paper, we introduce PreRL (Pre-train Space RL), which applies reward-driven online updates directly to P(y). We theoretically and empirically validate the strong gradient alignment between log P(y) and log P(y|x), establishing PreRL as a viable surrogate for standard RL. Furthermore, we uncover a critical mechanism: Negative Sample Reinforcement (NSR) within PreRL serves as an exceptionally effective driver for reasoning. NSR-PreRL rapidly prunes incorrect reasoning spaces while stimulating endogenous reflective behaviors, increasing transition and reflection thoughts by 14.89x and 6.54x, respectively. Leveraging these insights, we propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard RL for fine-grained optimization. Extensive experiments demonstrate that DSRL consistently outperforms strong baselines, proving that pre-train space pruning effectively steers the policy toward a refined correct reasoning subspace.
69e054eb5b704445ba50b733
Yuqiao Tan
69e054eb5b704445ba50b734
Minzheng Wang
claimed_verified
2026-04-16T07:59:18.375Z
64bcc373ef8c0e42bf16acc5
/avatars/873308203d28115ae1a9e4d0e26508f4.svg
mz.w
iiiiwis
user
iiiiwis
69e054eb5b704445ba50b735
Bo Liu
69e054eb5b704445ba50b736
Zichen Liu
69e054eb5b704445ba50b737
Tian Liang
69e054eb5b704445ba50b738
Shizhu He
69e054eb5b704445ba50b739
Jun Zhao
69e054eb5b704445ba50b73a
Kang Liu
From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space
While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model's existing output distribution. Optimizing the marginal distribution P(y) in the Pre-train Space addresses this bottleneck by encoding reasoning ability and preserving broad exploration capacity. Yet, conventional pre-training relies on static corpora for passive learning, leading to a distribution shift that hinders targeted reasoning enhancement. In this paper, we introduce PreRL (Pre-train Space RL), which applies reward-driven online updates directly to P(y). We theoretically and empirically validate the strong gradient alignment between log P(y) and log P(y|x), establishing PreRL as a viable surrogate for standard RL. Furthermore, we uncover a critical mechanism: Negative Sample Reinforcement (NSR) within PreRL serves as an exceptionally effective driver for reasoning. NSR-PreRL rapidly prunes incorrect reasoning spaces while stimulating endogenous reflective behaviors, increasing transition and reflection thoughts by 14.89x and 6.54x, respectively. Leveraging these insights, we propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard RL for fine-grained optimization. Extensive experiments demonstrate that DSRL consistently outperforms strong baselines, proving that pre-train space pruning effectively steers the policy toward a refined correct reasoning subspace.
PreRL applies reward-driven online updates to the marginal distribution in pre-train space, while DSRL uses NSR-PreRL to expand reasoning horizons before standard RL fine-tuning.
reinforcement learning with verifiable rewards conditional distribution marginal distribution pre-train space reward-driven online updates gradient alignment negative sample reinforcement policy reincarnation reasoning horizon standard RL