Black-Box On-Policy Distillation of Large Language Models
Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.
691708aab63bfc66e0498796
Tianzhu Ye
claimed_verified
2025-11-17T10:31:13.231Z
68ef6325918a07690443c922
/avatars/6200c248f89afca2bab5b62428f3df0d.svg
Tianzhu Ye
6eo
user
6eo
691708aab63bfc66e0498797
Li Dong
claimed_verified
2025-11-14T14:22:15.496Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
unilm
user
unilm
691708aab63bfc66e0498798
Zewen Chi
691708aab63bfc66e0498799
Xun Wu
691708aab63bfc66e049879a
Shaohan Huang
691708aab63bfc66e049879b
Furu Wei
Black-Box On-Policy Distillation of Large Language Models
Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.
Generative Adversarial Distillation (GAD) enhances black-box distillation by framing the student model as a generator and using a discriminator to provide adaptive feedback, surpassing traditional sequence-level knowledge distillation.
black-box distillation large language models (LLMs) Generative Adversarial Distillation (GAD) generator discriminator minimax game on-policy reward model sequence-level knowledge distillation LMSYS-Chat automatic evaluation