Beyond Correctness: Learning Robust Reasoning via Transfer
Reinforcement Learning with Verifiable Rewards (RLVR) has recently strengthened LLM reasoning, but its focus on final answer correctness leaves a critical gap: it does not ensure the robustness of the reasoning process itself. We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it, and treat reasoning as a form of meaning transfer that must survive truncation, reinterpretation, and continuation. Building on this principle, we introduce Reinforcement Learning with Transferable Reward (RLTR), which operationalizes robustness via transfer reward that tests whether a partial reasoning prefix from one model can guide a separate model to the correct answer. This encourages LLMs to produce reasoning that is stable, interpretable, and genuinely generalizable. Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps. For example, on MATH500, RLTR achieves a +3.6%p gain in Maj@64 compared to RLVR and matches RLVR's average accuracy with roughly 2.5x fewer training steps, providing both more reliable reasoning and significantly more sample efficient.
698c8152cea6ff55f58f0837
Hyunseok Lee
claimed_verified
2026-02-11T22:16:28.996Z
64b776ce81fbedb3938e7c0f
/avatars/684ad0a998acd9266e988fffbf396a4c.svg
HyunseokLee
hyunseoki
user
hyunseoki
698c8152cea6ff55f58f0838
Soheil Abbasloo
698c8152cea6ff55f58f0839
Jihoon Tack
698c8152cea6ff55f58f083a
Jinwoo Shin
Beyond Correctness: Learning Robust Reasoning via Transfer
Reinforcement Learning with Verifiable Rewards (RLVR) has recently strengthened LLM reasoning, but its focus on final answer correctness leaves a critical gap: it does not ensure the robustness of the reasoning process itself. We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it, and treat reasoning as a form of meaning transfer that must survive truncation, reinterpretation, and continuation. Building on this principle, we introduce Reinforcement Learning with Transferable Reward (RLTR), which operationalizes robustness via transfer reward that tests whether a partial reasoning prefix from one model can guide a separate model to the correct answer. This encourages LLMs to produce reasoning that is stable, interpretable, and genuinely generalizable. Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps. For example, on MATH500, RLTR achieves a +3.6%p gain in Maj@64 compared to RLVR and matches RLVR's average accuracy with roughly 2.5x fewer training steps, providing both more reliable reasoning and significantly more sample efficient.
Reinforcement Learning with Transferable Reward (RLTR) enhances LLM reasoning robustness by ensuring reasoning stability and generalizability through transfer rewards that test cross-model guidance capabilities.
Reinforcement Learning with Verifiable Rewards Reinforcement Learning with Transferable Reward LLM reasoning transfer reward reasoning robustness cross-model guidance sampling consistency final answer accuracy MATH500 Maj@64