MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD{https://github.com/top-yun/MAD}
697c3180a67238fac88cc175
Sangyun Chung
claimed_verified
2026-01-30T09:36:12.512Z
65a4bf8e90b5e87bcdff41c7
https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
Sangyun Chung
topyun
user
topyun
697c3180a67238fac88cc176
Se Yeon Kim
697c3180a67238fac88cc177
Youngchae Chee
697c3180a67238fac88cc178
Yong Man Ro
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD{https://github.com/top-yun/MAD}
Multimodal Large Language Models suffer from cross-modal hallucinations where one modality incorrectly influences generation from another, leading to fabricated outputs; this exposes a fundamental deficiency in modality-interaction control. To address this, a training-free method called Modality-Adaptive Decoding (MAD) is proposed that adaptively weights modality-specific decoding branches based on task requirements by leveraging the model's inherent ability to self-assess modality relevance. MAD uses extracted modality probabilities to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models, showing that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning.
Multimodal Large Language Models cross-modal hallucinations modality-interaction control Modality-Adaptive Decoding contrastive decoding modality-specific decoding branches self-assessment modality probabilities