Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces. In long-horizon settings, however, the common sampled-token variant is fragile: it reduces distribution matching to a one-token signal and becomes increasingly unreliable as rollouts drift away from prefixes the teacher commonly visits. We revisit OPD from the estimator and implementation sides. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL, but it has a much tighter worst-case variance bound; our toy study shows the same tradeoff empirically, with stronger future-reward coupling producing higher gradient variance and less stable learning. Empirically, we identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch. We address these issues with teacher top-K local support matching, implemented as truncated reverse-KL with top-p rollout sampling and special-token masking. Across single-task math reasoning and multi-task agentic-plus-math training, this objective yields more stable optimization and better downstream performance than sampled-token OPD.
69c649522de2ee20094ede73
Yuqian Fu
claimed_verified
2026-03-27T11:32:30.898Z
670aa09d35918e99fe7ff6b1
/avatars/5cbea2284165191e96544bacf2bfb50f.svg
Yuqian Fu
Yuqian-Fu
user
Yuqian-Fu
69c649522de2ee20094ede74
Haohuan Huang
claimed_verified
2026-03-30T08:59:09.479Z
6860d8e658095f461a850ff3
https://cdn-avatars.huggingface.co/v1/production/uploads/6860d8e658095f461a850ff3/BpOMOqf5B0s9e_8hVHy6W.jpeg
Haohuan Huang
hhh675597
user
hhh675597
69c649522de2ee20094ede75
Kaiwen Jiang
69c649522de2ee20094ede76
Yuanheng Zhu
69c649522de2ee20094ede77
Dongbin Zhao
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces. In long-horizon settings, however, the common sampled-token variant is fragile: it reduces distribution matching to a one-token signal and becomes increasingly unreliable as rollouts drift away from prefixes the teacher commonly visits. We revisit OPD from the estimator and implementation sides. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL, but it has a much tighter worst-case variance bound; our toy study shows the same tradeoff empirically, with stronger future-reward coupling producing higher gradient variance and less stable learning. Empirically, we identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch. We address these issues with teacher top-K local support matching, implemented as truncated reverse-KL with top-p rollout sampling and special-token masking. Across single-task math reasoning and multi-task agentic-plus-math training, this objective yields more stable optimization and better downstream performance than sampled-token OPD.
On-policy distillation for large language models faces challenges in long-horizon settings due to token-level signal fragility, which is addressed through improved estimation methods and implementation techniques.
on-policy distillation large language models teacher feedback student-generated rollouts sampled-token variant distribution matching reverse-KL gradient variance token-level estimation sequence-level estimation top-K local support matching top-p rollout sampling special-token masking