Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.
693296da6d1060ca587a2812
Tianle Chen
693296da6d1060ca587a2813
Chaitanya Chakka
693296da6d1060ca587a2814
Arjun Reddy Akula
693296da6d1060ca587a2815
Xavier Thomas
claimed_verified
2026-04-13T07:39:38.771Z
6505b787dacc94cd6ce9d256
/avatars/9490807f97afc39aeb663d343a900ff9.svg
Thomas
XThomasBU
user
XThomasBU
693296da6d1060ca587a2816
Deepti Ghadiyaram
claimed_verified
2026-04-13T07:39:40.536Z
6905068e4c9138632af2fa21
/avatars/c32215347601de938c218e6c81dc2e33.svg
Deepti Ghadiyaram
dghadiya
user
dghadiya
Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.
Multimodal Large Language Models (MLLMs) lack robustness to contradictory modalities, as demonstrated by MMA-Bench, and a modality alignment tuning strategy improves their multimodal reasoning.
Multimodal Large Language Models MLLMs MMA-Bench black-box interpretability white-box interpretability modality alignment tuning multimodal reasoning misaligned audio-visual pairs misleading text multimodal grounding