Do Audio-Visual Large Language Models Really See and Hear?
Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
69d5141f63f089495f06b22f
Ramaneswaran Selvakumar
69d5141f63f089495f06b230
Kaousheik Jayakumar
69d5141f63f089495f06b231
S Sakshi
69d5141f63f089495f06b232
Sreyan Ghosh
claimed_verified
2026-04-16T08:19:04.718Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
Sreyan88
user
Sreyan88
69d5141f63f089495f06b233
Ruohan Gao
69d5141f63f089495f06b234
Dinesh Manocha
Do Audio-Visual Large Language Models Really See and Hear?
Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
AVLLMs exhibit modality bias where visual representations dominate over audio cues during multimodal integration, despite audio semantics being present in intermediate layers.
Audio-Visual Large Language Models mechanistic interpretability multimodal perception audio-visual fusion modality bias vision-language base model audio semantics visual representations text generation