Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.
69b104cd1f7902d47f6dfab4
Zhengzhao Ma
69b104cd1f7902d47f6dfab5
Xueru Wen
69b104cd1f7902d47f6dfab6
Boxi Cao
69b104cd1f7902d47f6dfab7
Yaojie Lu
69b104cd1f7902d47f6dfab8
Hongyu Lin
69b104cd1f7902d47f6dfab9
Jinglin Yang
69b104cd1f7902d47f6dfaba
Min He
69b104cd1f7902d47f6dfabb
Xianpei Han
69b104cd1f7902d47f6dfabc
Le Sun
Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.
DCPO framework decouples reasoning and calibration objectives in LLMs to address calibration degeneration while maintaining high accuracy.
reinforcement learning from verifiable rewards large language models calibration degeneration policy accuracy calibration error gradient conflict direct comparison policy optimization GRPO