From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
Reinforcement learning has emerged as a paradigm for post-training large language models, boosting their reasoning capabilities. Such approaches compute an advantage value for each sample, reflecting better or worse performance than expected, thereby yielding both positive and negative signals for training. However, the indiscriminate mixing of the two signals in existing methods, especially from the early stages, may lead to ambiguous guidance and limited gains. To address this issue, we propose **CAPO** (**C**urriculum **A**dvantage **P**olicy **O**ptimization), an adaptive curriculum mechanism based on advantage signals. The proposed mechanism bootstraps imitation learning with positive-only advantage samples to establish robust foundations, and subsequently introduces negative signals to cultivate discriminative capabilities, thereby improving generalization across complex scenarios. Compatible with diverse optimization methods including GRPO, PPO, RLOO, and Reinforce++, our method consistently achieves stable and significant improvements in mathematical reasoning tasks, and further generalizes effectively to multimodal Graphical User Interface (GUI) reasoning scenarios, establishing itself as a versatile and robust optimization framework.
6934d2ba3962c926cf680575
Changpeng Yang
6934d2ba3962c926cf680576
Jinyang Wu
6934d2ba3962c926cf680577
Yuchen Liu
6934d2ba3962c926cf680578
Shuai Zhang
6934d2ba3962c926cf680579
Yang Li
6934d2ba3962c926cf68057a
Qiliang Liang
6934d2ba3962c926cf68057b
Hongzhen Wang
6934d2ba3962c926cf68057c
Shuai Nie
6934d2ba3962c926cf68057d
Jiaming Xu
6934d2ba3962c926cf68057e
Runyu Shi
6934d2ba3962c926cf68057f
Ying Huang
6934d2ba3962c926cf680580
Guoquan Zhang
From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
Reinforcement learning has emerged as a paradigm for post-training large language models, boosting their reasoning capabilities. Such approaches compute an advantage value for each sample, reflecting better or worse performance than expected, thereby yielding both positive and negative signals for training. However, the indiscriminate mixing of the two signals in existing methods, especially from the early stages, may lead to ambiguous guidance and limited gains. To address this issue, we propose **CAPO** (**C**urriculum **A**dvantage **P**olicy **O**ptimization), an adaptive curriculum mechanism based on advantage signals. The proposed mechanism bootstraps imitation learning with positive-only advantage samples to establish robust foundations, and subsequently introduces negative signals to cultivate discriminative capabilities, thereby improving generalization across complex scenarios. Compatible with diverse optimization methods including GRPO, PPO, RLOO, and Reinforce++, our method consistently achieves stable and significant improvements in mathematical reasoning tasks, and further generalizes effectively to multimodal Graphical User Interface (GUI) reasoning scenarios, establishing itself as a versatile and robust optimization framework.
CAPO, a curriculum advantage policy optimization, enhances reinforcement learning for large language models by strategically introducing positive and negative advantage signals, improving reasoning capabilities and generalization.
reinforcement learning post-training large language models reasoning capabilities advantage value positive signals negative signals curriculum mechanism imitation learning discriminative capabilities generalization GRPO PPO RLOO Reinforce++ mathematical reasoning tasks multimodal Graphical User Interface (GUI) reasoning scenarios optimization framework