Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.
69e19335860e47828575b3b2
Haoyi Sun
claimed_verified
2026-04-17T06:56:58.550Z
64d09f110b71aea8be7fe14d
/avatars/3944b564d4c5cdecfc3c5b88dd7adb0d.svg
HaoyiSun
HaoyiSun
user
HaoyiSun
69e19335860e47828575b3b3
Xiaoxiao Wang
69e19335860e47828575b3b4
Ning Mao
69e19335860e47828575b3b5
Qian Wang
69e19335860e47828575b3b6
Lifu Mu
69e19335860e47828575b3b7
Wen Zheng
69e19335860e47828575b3b8
Tao Wei
69e19335860e47828575b3b9
Wei Chen
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.
Vision-language models face deployment challenges due to their large size, but knowledge distillation can improve efficiency while maintaining performance through a novel visual-switch framework that enhances multimodal knowledge transfer.
Vision-Language Models Knowledge Distillation multimodal knowledge transfer visual-switch distillation cross-modal probabilistic references dynamic bi-directional logits difference loss teacher-student framework multimodal alignment