Efficient RLVR Training via Weighted Mutual Information Data Selection
Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.
69a68874663a122a7ce668a1
Xinyu Zhou
claimed_verified
2026-03-03T10:36:35.622Z
6469e4ac4c1cd18b497537bb
/avatars/5149203a9015956578deaf3710c30cef.svg
Zhou
xinyu04
user
xinyu04
69a68874663a122a7ce668a2
Boyu Zhu
69a68874663a122a7ce668a3
Haotian Zhang
69a68874663a122a7ce668a4
Huiming Wang
69a68874663a122a7ce668a5
Zhijiang Guo
Efficient RLVR Training via Weighted Mutual Information Data Selection
Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.
InSight is an information-guided data sampling method for reinforcement learning training that improves efficiency by considering both difficulty and epistemic uncertainty through Bayesian modeling and weighted mutual information objectives.
reinforcement learning difficulty-based heuristics epistemic uncertainty Bayesian latent success rates weighted mutual information acquisition score multi-rollout settings reinforcement learning with verifiable rewards planning benchmarks mathematics benchmarks general reasoning