Data-Efficient RLVR via Off-Policy Influence Guidance
Data selection is a critical aspect of Reinforcement Learning with Verifiable
Rewards (RLVR) for enhancing the reasoning capabilities of large language
models (LLMs). Current data selection methods are largely heuristic-based,
lacking theoretical guarantees and generalizability. This work proposes a
theoretically-grounded approach using influence functions to estimate the
contribution of each data point to the learning objective. To overcome the
prohibitive computational cost of policy rollouts required for online influence
estimation, we introduce an off-policy influence estimation method that
efficiently approximates data influence using pre-collected offline
trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we
employ sparse random projection to reduce dimensionality and improve storage
and computation efficiency. Leveraging these techniques, we develop
Curriculum RL with Off-Policy
Influence guidance (CROPI), a multi-stage RL framework that
iteratively selects the most influential data for the current policy.
Experiments on models up to 7B parameters demonstrate that CROPI significantly
accelerates training. On a 1.5B model, it achieves a 2.66x step-level
acceleration while using only 10\% of the data per stage compared to
full-dataset training. Our results highlight the substantial potential of
influence-based data selection for efficient RLVR.
6905baa1b26ae8b17699fd58
Erle Zhu
claimed_verified
2025-11-03T20:53:52.262Z
6421c749eaad1bcb28b11206
https://cdn-avatars.huggingface.co/v1/production/uploads/6421c749eaad1bcb28b11206/jvaBUG7Z5fv6lJMoH301q.jpeg
Erle Zhu
lez3f
user
lez3f
6905baa1b26ae8b17699fd59
Dazhi Jiang
6905baa1b26ae8b17699fd5a
Yuan Wang
claimed_verified
2025-11-04T20:27:15.733Z
6864ef64d4dde09fbabad95a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sbEmJxiJg4-8rrXDqUuFy.jpeg
Yuan Wang
traveler2333
user
traveler2333
6905baa1b26ae8b17699fd5b
Xujun Li
6905baa1b26ae8b17699fd5c
Jiale Cheng
6905baa1b26ae8b17699fd5d
Yuxian Gu
6905baa1b26ae8b17699fd5e
Yilin Niu
6905baa1b26ae8b17699fd5f
Aohan Zeng
6905baa1b26ae8b17699fd60
Jie Tang
6905baa1b26ae8b17699fd61
Minlie Huang
6905baa1b26ae8b17699fd62
Hongning Wang
Data-Efficient RLVR via Off-Policy Influence Guidance
Data selection is a critical aspect of Reinforcement Learning with Verifiable
Rewards (RLVR) for enhancing the reasoning capabilities of large language
models (LLMs). Current data selection methods are largely heuristic-based,
lacking theoretical guarantees and generalizability. This work proposes a
theoretically-grounded approach using influence functions to estimate the
contribution of each data point to the learning objective. To overcome the
prohibitive computational cost of policy rollouts required for online influence
estimation, we introduce an off-policy influence estimation method that
efficiently approximates data influence using pre-collected offline
trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we
employ sparse random projection to reduce dimensionality and improve storage
and computation efficiency. Leveraging these techniques, we develop
Curriculum RL with Off-Policy
Influence guidance (CROPI), a multi-stage RL framework that
iteratively selects the most influential data for the current policy.
Experiments on models up to 7B parameters demonstrate that CROPI significantly
accelerates training. On a 1.5B model, it achieves a 2.66x step-level
acceleration while using only 10\% of the data per stage compared to
full-dataset training. Our results highlight the substantial potential of
influence-based data selection for efficient RLVR.
Influence functions and off-policy estimation improve data selection in RLVR, accelerating training and reducing data usage for large language models.
Reinforcement Learning with Verifiable Rewards (RLVR) influence functions policy rollouts off-policy influence estimation sparse random projection Curriculum RL with Off-Policy Influence guidance (CROPI) high-dimensional gradients multi-stage RL framework