Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.
69def4b2d3c84fcbf292159a
Yaxuan Li
claimed_verified
2026-04-15T11:51:00.520Z
66955fc5bd8faee706e490e7
/avatars/4ce826f2926cf736e3b93ee2fd8edb51.svg
liyaxuan
lllyx
user
lllyx
69def4b2d3c84fcbf292159b
Yuxin Zuo
claimed_verified
2026-04-15T11:50:57.846Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
yuxinzuo
user
yuxinzuo
69def4b2d3c84fcbf292159c
Bingxiang He
claimed_verified
2026-04-16T08:03:35.271Z
64c5e944979493279b700cb2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vjFuPWw8Vl7b7gXB19Sk-.jpeg
Bingxiang He
hbx
user
hbx
69def4b2d3c84fcbf292159d
Jinqian Zhang
69def4b2d3c84fcbf292159e
Chaojun Xiao
69def4b2d3c84fcbf292159f
Cheng Qian
69def4b2d3c84fcbf29215a0
Tianyu Yu
69def4b2d3c84fcbf29215a1
Huan-ang Gao
69def4b2d3c84fcbf29215a2
Wenkai Yang
claimed_verified
2026-04-17T09:34:02.728Z
64b7df742f5a966b973e25f7
/avatars/e24e7769188d441317b3b7d10ef8fd60.svg
Wenkai Yang
Keven16
user
Keven16
69def4b2d3c84fcbf29215a3
Zhiyuan Liu
69def4b2d3c84fcbf29215a4
Ning Ding
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.
On-policy distillation dynamics in large language models depend on compatible thinking patterns between teacher and student models, with successful distillation characterized by alignment on high-probability tokens and requiring teachers to provide novel capabilities beyond student training data.
on-policy distillation large language models teacher-student framework token-level mechanism weak-to-strong reverse distillation distributional indistinguishability high-probability tokens token-level reward long-horizon distillation