CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.
69b21d79206bad330cd7ff44
Sijia Cui
claimed_verified
2026-03-23T11:00:21.382Z
658bd850c88fda265ce659cb
/avatars/c5a68a789745828e7f3aabee05bb1eff.svg
Sijia Cui
cuisijia
user
cuisijia
69b21d79206bad330cd7ff45
Pengyu Cheng
69b21d79206bad330cd7ff46
Jiajun Song
69b21d79206bad330cd7ff47
Yongbo Gai
69b21d79206bad330cd7ff48
Guojun Zhang
admin_assigned
2026-03-12T10:13:43.148Z
6511d02684bedffb5aa9509e
/avatars/ed4f546f4fef206a71227513995d4d72.svg
Guojun Zhang
gordon-z
user
gordon-z
69b21d79206bad330cd7ff49
Zhechao Yu
admin_assigned
2026-03-12T10:13:54.122Z
665e7762bbea8949182b1271
/avatars/53c59bb7011c6330a6cdc2898c430c3f.svg
Zhechao Yu
Marcoyu11
user
Marcoyu11
69b21d79206bad330cd7ff4a
Jianhe Lin
69b21d79206bad330cd7ff4b
Xiaoxi Jiang
69b21d79206bad330cd7ff4c
Guanjun Jiang
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.
Contrastive Learning mechanism integrated into Policy Optimization enhances LLM reasoning by regularizing correct reasoning paths and reducing hallucinations.
Reinforcement Learning with Verifiable Rewards Large Language Models Policy Optimization Contrastive Learning reasoning capacity hallucination answer-copying cross-trajectory regularization step-level reasoning inconsistencies