Rethinking Selective Knowledge Distillation
Growing efforts to improve knowledge distillation (KD) in large language models (LLMs) replace dense teacher supervision with selective distillation, which uses a subset of token positions, vocabulary classes, or training samples for supervision. However, it remains unclear which importance signals, selection policies, and their interplay are most effective. In this work, we revisit where and how to distill in autoregressive LLMs. We disentangle selective KD along the position, class, and sample axes and systematically compare importance signals and selection policies. Then, guided by this analysis, we identify underexplored opportunities and introduce student-entropy-guided position selection (SE-KD). Across a suite of benchmarks, SE-KD often improves accuracy, downstream task adherence, and memory efficiency over dense distillation. Extending this approach across the class and sample axes (SE-KD 3X) yields complementary efficiency gains that make offline teacher caching feasible. In practice, this reduces wall time by 70% and peak memory by 18%, while cutting storage usage by 80% over prior methods without sacrificing performance.
69818b3ace18b18628096395
Almog Tavor
claimed_verified
2026-02-03T10:03:04.206Z
6113ec1a570f75d1806eab09
https://cdn-avatars.huggingface.co/v1/production/uploads/6113ec1a570f75d1806eab09/3UrtWn_2OiVyYiB6NXEBX.jpeg
Almog Tavor
almogtavor
user
almogtavor
69818b3ace18b18628096396
Itay Ebenspanger
claimed_verified
2026-02-06T18:57:36.965Z
69853b9af62635603d840c6f
/avatars/9cea3ef49a8d4b8eb91c34747b6dceca.svg
Itay Ebenspanger
itayebens
user
itayebens
69818b3ace18b18628096397
Neil Cnaan
claimed_verified
2026-02-03T10:03:00.682Z
6330c041e7efa1d1a884b576
/avatars/4d5be3b7d2f2f6af97b99d266693a339.svg
Neil
N-il
user
N-il
69818b3ace18b18628096398
Mor Geva
claimed_verified
2026-02-03T13:58:15.704Z
610b729f9da682cd54ad9adf
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
Mor Geva
mega
user
mega
Rethinking Selective Knowledge Distillation
Growing efforts to improve knowledge distillation (KD) in large language models (LLMs) replace dense teacher supervision with selective distillation, which uses a subset of token positions, vocabulary classes, or training samples for supervision. However, it remains unclear which importance signals, selection policies, and their interplay are most effective. In this work, we revisit where and how to distill in autoregressive LLMs. We disentangle selective KD along the position, class, and sample axes and systematically compare importance signals and selection policies. Then, guided by this analysis, we identify underexplored opportunities and introduce student-entropy-guided position selection (SE-KD). Across a suite of benchmarks, SE-KD often improves accuracy, downstream task adherence, and memory efficiency over dense distillation. Extending this approach across the class and sample axes (SE-KD 3X) yields complementary efficiency gains that make offline teacher caching feasible. In practice, this reduces wall time by 70% and peak memory by 18%, while cutting storage usage by 80% over prior methods without sacrificing performance.
Selective knowledge distillation in autoregressive language models using student-entropy-guided position selection improves accuracy and efficiency while reducing memory and storage requirements.
knowledge distillation large language models selective distillation autoregressive models student-entropy-guided position selection dense distillation offline teacher caching