How Far Can Unsupervised RLVR Scale LLM Training?
Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.
69af9574b3ab31dee1281d64
Bingxiang He
69af9574b3ab31dee1281d65
Yuxin Zuo
claimed_verified
2026-04-15T12:09:38.042Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
yuxinzuo
user
yuxinzuo
69af9574b3ab31dee1281d66
Zeyuan Liu
69af9574b3ab31dee1281d67
Shangziqi Zhao
69af9574b3ab31dee1281d68
Zixuan Fu
69af9574b3ab31dee1281d69
Junlin Yang
claimed_verified
2026-03-10T09:44:53.845Z
66ed083acaf696884760729a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RgPe99BqsJsHUWoXO1qtS.jpeg
Nick Yang
RadioBlue
user
RadioBlue
69af9574b3ab31dee1281d6a
Cheng Qian
69af9574b3ab31dee1281d6b
Kaiyan Zhang
69af9574b3ab31dee1281d6c
Yuchen Fan
69af9574b3ab31dee1281d6d
Ganqu Cui
69af9574b3ab31dee1281d6e
Xiusi Chen
69af9574b3ab31dee1281d6f
Youbang Sun
69af9574b3ab31dee1281d70
Xingtai Lv
claimed_verified
2026-03-11T08:45:11.854Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
XingtaiHF
user
XingtaiHF
69af9574b3ab31dee1281d71
Xuekai Zhu
69af9574b3ab31dee1281d72
Li Sheng
69af9574b3ab31dee1281d73
Ran Li
69af9574b3ab31dee1281d74
Huan-ang Gao
69af9574b3ab31dee1281d75
Yuchen Zhang
69af9574b3ab31dee1281d76
Bowen Zhou
69af9574b3ab31dee1281d77
Zhiyuan Liu
69af9574b3ab31dee1281d78
Ning Ding
How Far Can Unsupervised RLVR Scale LLM Training?
Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.
Unsupervised reinforcement learning with verifiable rewards faces fundamental limitations in scaling large language model training due to inherent convergence properties and confidence-correction misalignment, though external methods show promise for overcoming these barriers.
unsupervised reinforcement learning verifiable rewards large language model training intrinsic signals reward derivation theoretical framework distribution sharpening model prior test-time training computational asymmetries