Reinforcement Learning via Self-Distillation
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.
697b378a870173bd9177734d
Jonas Hübotter
claimed_verified
2026-01-29T13:56:34.106Z
6645dbab80ee4eb5ddc8aed8
https://cdn-avatars.huggingface.co/v1/production/uploads/6645dbab80ee4eb5ddc8aed8/-IckaooWMXeACKew9THv_.jpeg
Jonas Hübotter
jonhue
user
jonhue
697b378a870173bd9177734e
Frederike Lübeck
697b378a870173bd9177734f
Lejs Behric
697b378a870173bd91777350
Anton Baumann
claimed_verified
2026-02-11T11:20:42.303Z
631da3612899cb403efab153
/avatars/0a081c2aa92ad7e23d75c7ef2554e174.svg
Anton Baumann
antonbaumann
user
antonbaumann
697b378a870173bd91777351
Marco Bagatella
697b378a870173bd91777352
Daniel Marta
697b378a870173bd91777353
Ido Hakimi
697b378a870173bd91777354
Idan Shenfeld
697b378a870173bd91777355
Thomas Kleine Buening
697b378a870173bd91777356
Carlos Guestrin
697b378a870173bd91777357
Andreas Krause
Reinforcement Learning via Self-Distillation
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.
Self-Distillation Policy Optimization (SDPO) enhances reinforcement learning with verifiable rewards by utilizing rich textual feedback to improve sample efficiency and accuracy in language model training.
reinforcement learning verifiable rewards rich feedback Self-Distillation Policy Optimization SDPO tokenized feedback dense learning signal reward model policy distillation in-context learning sample efficiency accuracy