Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6times speedup. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.
698171fbce18b18628096024
Yiju Guo
claimed_verified
2026-02-03T10:07:34.331Z
64bb937d8496ee0fb6cac9aa
https://cdn-avatars.huggingface.co/v1/production/uploads/64bb937d8496ee0fb6cac9aa/oFkKNxaMrd3wAciwP4Lu5.png
YijuGuo
YijuGuo
user
YijuGuo
698171fbce18b18628096025
Tianyi Hu
698171fbce18b18628096026
Zexu Sun
claimed_verified
2026-02-11T11:20:20.652Z
65d1b42f3da87ce21e33261a
/avatars/041cb441fa3871acde4ba565632056bf.svg
RubinSun
RubinSun
user
RubinSun
698171fbce18b18628096027
Yankai Lin
Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6times speedup. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.
LENS framework improves reinforcement learning with verifiable rewards by identifying and removing interference tokens to enhance exploration efficiency and training stability.
Reinforcement Learning with Verifiable Rewards exploration rollout budget sampling success policy optimization interference tokens prompt tokens GRPO LENS