Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of Knowledge Purification, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models.
69895b69beecc443208d26ce
Ruihan Jin
69895b69beecc443208d26cf
Pengpeng Shao
69895b69beecc443208d26d0
Zhengqi Wen
69895b69beecc443208d26d1
Jinyang Wu
claimed_verified
2026-02-09T08:29:25.039Z
6747de57f8cab58c22ec94a2
/avatars/5bae0341862fac24564781c0fa32aac5.svg
Jinyang Wu
Jinyang23
user
Jinyang23
69895b69beecc443208d26d2
Mingkuan Feng
69895b69beecc443208d26d3
Shuo Yang
69895b69beecc443208d26d4
Chu Yuan Zhang
69895b69beecc443208d26d5
Jianhua Tao
Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of Knowledge Purification, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models.
Knowledge purification techniques consolidate rationales from multiple teacher LLMs to reduce conflicts and improve efficiency in distillation processes.
knowledge distillation large language models teacher models knowledge conflicts knowledge purification router-based methods