LinguDistill: Recovering Linguistic Ability in Vision- Language Models via Selective Cross-Modal Distillation
Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representation shift and cross-modal interference introduced during multimodal adaptation. Such loss is difficult to recover, even with targeted task-specific fine-tuning using standard objectives. Prior recovery approaches typically introduce additional modules that act as intermediate alignment layers to maintain or isolate modality-specific subspaces, which increases architectural complexity, adds parameters at inference time, and limits flexibility across models and settings. We propose LinguDistill, an adapter-free distillation method that restores linguistic capability by utilizing the original frozen LM as a teacher. We overcome the key challenge of enabling vision-conditioned teacher supervision by introducing layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without modifying the architecture of either model. We then selectively distill the teacher's strong linguistic signal on language-intensive data to recover language capability, while preserving the student's visual grounding on multimodal tasks. As a result, LinguDistill recovers sim10% of the performance lost on language and knowledge benchmarks, while maintaining comparable performance on vision-heavy tasks. Our findings demonstrate that linguistic capability can be recovered without additional modules, providing an efficient and practical solution to modality-specific degradation in multimodal models.
69cf442de02fdd68b91b7ae6
Patrick Amadeus Irawan
claimed_verified
2026-04-03T09:07:38.496Z
64d1e3a87e20ec9ea0020d03
https://cdn-avatars.huggingface.co/v1/production/uploads/64d1e3a87e20ec9ea0020d03/xm-afh1AaqS0e9qpaPo7y.jpeg
Patrick Amadeus Irawan
patrickamadeus
user
patrickamadeus
69cf442de02fdd68b91b7ae7
Erland Hilman Fuadi
claimed_verified
2026-04-07T08:45:23.058Z
6239ac869895e5c2a4345131
https://cdn-avatars.huggingface.co/v1/production/uploads/6239ac869895e5c2a4345131/QErVhX2EdKUSg06eRrU7L.jpeg
Edd
Erland
user
Erland
69cf442de02fdd68b91b7ae8
Shanu Kumar
69cf442de02fdd68b91b7ae9
Alham Fikri Aji
69cf442de02fdd68b91b7aea
Yova Kementchedjhieva
LinguDistill: Recovering Linguistic Ability in Vision- Language Models via Selective Cross-Modal Distillation
Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representation shift and cross-modal interference introduced during multimodal adaptation. Such loss is difficult to recover, even with targeted task-specific fine-tuning using standard objectives. Prior recovery approaches typically introduce additional modules that act as intermediate alignment layers to maintain or isolate modality-specific subspaces, which increases architectural complexity, adds parameters at inference time, and limits flexibility across models and settings. We propose LinguDistill, an adapter-free distillation method that restores linguistic capability by utilizing the original frozen LM as a teacher. We overcome the key challenge of enabling vision-conditioned teacher supervision by introducing layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without modifying the architecture of either model. We then selectively distill the teacher's strong linguistic signal on language-intensive data to recover language capability, while preserving the student's visual grounding on multimodal tasks. As a result, LinguDistill recovers sim10% of the performance lost on language and knowledge benchmarks, while maintaining comparable performance on vision-heavy tasks. Our findings demonstrate that linguistic capability can be recovered without additional modules, providing an efficient and practical solution to modality-specific degradation in multimodal models.
LinguDistill enables recovery of linguistic capabilities in vision-language models through adapter-free distillation using frozen language models as teachers, achieving performance close to pre-adaptation levels while maintaining visual task performance.
pretrained language models vision-language models representation shift cross-modal interference distillation KV-cache sharing multimodal representations language-intensive data visual grounding