Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization.
69e1382d4bed1847510d9346
Yinghui He
69e1382d4bed1847510d9347
Simran Kaur
69e1382d4bed1847510d9348
Adithya Bhaskar
69e1382d4bed1847510d9349
Yongjin Yang
69e1382d4bed1847510d934a
Jiarui Liu
69e1382d4bed1847510d934b
Narutatsu Ri
claimed_verified
2026-04-16T20:20:16.525Z
64698ed0dcbb937d56b9dd02
/avatars/407afd70356f9d0669d4e2d39b9a8740.svg
Narutatsu Ri
narutatsuri
user
narutatsuri
69e1382d4bed1847510d934c
Liam Fowl
69e1382d4bed1847510d934d
Abhishek Panigrahi
69e1382d4bed1847510d934e
Danqi Chen
69e1382d4bed1847510d934f
Sanjeev Arora
Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization.
Self-Distillation Zero trains a model to transform binary rewards into dense token-level self-supervision through dual-role training and on-policy self-distillation, achieving superior performance in reasoning tasks with reduced sample efficiency requirements.
reinforcement learning distillation self-distillation binary rewards token-level supervision on-policy self-distillation token-level self-localization iterative self-evolution teacher synchronization