Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning abilities of large language models (LLMs), yet training often stalls as problems become saturated. We identify the core challenge as the poor accessibility of informative failures: learning signals exist but are rarely encountered during standard rollouts. To address this, we propose failure-prefix conditioning, a simple and effective method for learning from saturated problems. Rather than starting from the original question, our approach reallocates exploration by conditioning training on prefixes derived from rare incorrect reasoning trajectories, thereby exposing the model to failure-prone states. We observe that failure-prefix conditioning yields performance gains matching those of training on medium-difficulty problems, while preserving token efficiency. Furthermore, we analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results suggest that failure-prefix conditioning offers an effective pathway to extend RLVR training on saturated problems.
697b7b4601afa04094ca0864
Minwu Kim
claimed_verified
2026-02-02T17:00:36.624Z
64e77b47d96966317b45eeb3
/avatars/6b67eba3f15d6cd86ac3ad55c1daf166.svg
Minwu Kim
guactastesgood
user
guactastesgood
697b7b4601afa04094ca0865
Safal Shrestha
claimed_verified
2026-02-03T10:09:07.288Z
64cb922ec7f30fbf7b91a9a7
/avatars/457eae5e56b9641ee5543146447d1755.svg
Safal Shrestha
safal312
user
safal312
697b7b4601afa04094ca0866
Keith Ross
Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning abilities of large language models (LLMs), yet training often stalls as problems become saturated. We identify the core challenge as the poor accessibility of informative failures: learning signals exist but are rarely encountered during standard rollouts. To address this, we propose failure-prefix conditioning, a simple and effective method for learning from saturated problems. Rather than starting from the original question, our approach reallocates exploration by conditioning training on prefixes derived from rare incorrect reasoning trajectories, thereby exposing the model to failure-prone states. We observe that failure-prefix conditioning yields performance gains matching those of training on medium-difficulty problems, while preserving token efficiency. Furthermore, we analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results suggest that failure-prefix conditioning offers an effective pathway to extend RLVR training on saturated problems.
Failure-prefix conditioning enables more effective reinforcement learning from saturated problems by focusing exploration on informative failure trajectories, maintaining token efficiency while improving robustness.
reinforcement learning large language models reasoning abilities training stagnation informative failures failure-prefix conditioning exploration token efficiency robustness iterative approach saturated problems