Memorization Dynamics in Knowledge Distillation for Language Models
Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility while often surpassing standard fine-tuning. Beyond performance, KD is also explored as a privacy-preserving mechanism to mitigate the risk of training data leakage. While training data memorization has been extensively studied in standard pre-training and fine-tuning settings, its dynamics in a knowledge distillation setup remain poorly understood. In this work, we study memorization across the KD pipeline using three large language model (LLM) families (Pythia, OLMo-2, Qwen-3) and three datasets (FineWeb, Wikitext, Nemotron-CC-v2). We find: (1) distilled models memorize significantly less training data than standard fine-tuning (reducing memorization by more than 50%); (2) some examples are inherently easier to memorize and account for a large fraction of memorization during distillation (over ~95%); (3) student memorization is predictable prior to distillation using features based on zlib entropy, KL divergence, and perplexity; and (4) while soft and hard distillation have similar overall memorization rates, hard distillation poses a greater risk: it inherits 2.7times more teacher-specific examples than soft distillation. Overall, we demonstrate that distillation can provide both improved generalization and reduced memorization risks compared to standard fine-tuning.
697d7a2e6676f9332270621b
Jaydeep Borkar
claimed_verified
2026-02-02T16:57:50.378Z
63c38f888cc87cf0c06abfe9
/avatars/e99b662404fb805870d95e0f1f17167b.svg
Jay
jaydeepb
user
jaydeepb
697d7a2e6676f9332270621c
Karan Chadha
697d7a2e6676f9332270621d
Niloofar Mireshghallah
697d7a2e6676f9332270621e
Yuchen Zhang
697d7a2e6676f9332270621f
Irina-Elena Veliche
697d7a2e6676f93322706220
Archi Mitra
697d7a2e6676f93322706221
David A. Smith
697d7a2e6676f93322706222
Zheng Xu
697d7a2e6676f93322706223
Diego Garcia-Olano
Memorization Dynamics in Knowledge Distillation for Language Models
Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility while often surpassing standard fine-tuning. Beyond performance, KD is also explored as a privacy-preserving mechanism to mitigate the risk of training data leakage. While training data memorization has been extensively studied in standard pre-training and fine-tuning settings, its dynamics in a knowledge distillation setup remain poorly understood. In this work, we study memorization across the KD pipeline using three large language model (LLM) families (Pythia, OLMo-2, Qwen-3) and three datasets (FineWeb, Wikitext, Nemotron-CC-v2). We find: (1) distilled models memorize significantly less training data than standard fine-tuning (reducing memorization by more than 50%); (2) some examples are inherently easier to memorize and account for a large fraction of memorization during distillation (over ~95%); (3) student memorization is predictable prior to distillation using features based on zlib entropy, KL divergence, and perplexity; and (4) while soft and hard distillation have similar overall memorization rates, hard distillation poses a greater risk: it inherits 2.7times more teacher-specific examples than soft distillation. Overall, we demonstrate that distillation can provide both improved generalization and reduced memorization risks compared to standard fine-tuning.
Knowledge distillation reduces training data memorization compared to standard fine-tuning while maintaining performance, with distinct memorization patterns and predictability based on input characteristics.
knowledge distillation large language models memorization soft distillation hard distillation zlib entropy KL divergence perplexity