On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference Δlog p between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that Δlog p more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a test-time extrapolation method that amplifies the policy along the learned Δlog p direction to improve reasoning accuracy without further training; (2) a training-time reweighting method that focuses learning on low-probability (corresponding to higher Δlog p) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.
69c2036cc105f45cb0481e9e
Kexin Huang
claimed_verified
2026-03-30T09:10:43.513Z
6547a247dbce6bd2be168d33
/avatars/23ae384f7c4fc1a89a556a37f5e75acf.svg
Kexin Huang
737443h
user
737443h
69c2036cc105f45cb0481e9f
Haoming Meng
69c2036cc105f45cb0481ea0
Junkang Wu
69c2036cc105f45cb0481ea1
Jinda Lu
69c2036cc105f45cb0481ea2
Chiyu Ma
69c2036cc105f45cb0481ea3
Ziqian Chen
69c2036cc105f45cb0481ea4
Xue Wang
69c2036cc105f45cb0481ea5
Bolin Ding
69c2036cc105f45cb0481ea6
Jiancan Wu
69c2036cc105f45cb0481ea7
Xiang Wang
69c2036cc105f45cb0481ea8
Xiangnan He
69c2036cc105f45cb0481ea9
Guoyin Wang
69c2036cc105f45cb0481eaa
Jingren Zhou
On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference Δlog p between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that Δlog p more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a test-time extrapolation method that amplifies the policy along the learned Δlog p direction to improve reasoning accuracy without further training; (2) a training-time reweighting method that focuses learning on low-probability (corresponding to higher Δlog p) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.
Reinforcement learning with verifiable rewards improves language model reasoning by focusing on the direction of parameter updates rather than their magnitude, enabling better test-time extrapolation and training-time reweighting methods.
reinforcement learning with verifiable rewards language models token-level log probability difference test-time extrapolation training-time reweighting policy amplification reasoning accuracy