When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs
  Preference Dynamics in MLLMs
Multimodal large language models (MLLMs) must resolve conflicts when
different modalities provide contradictory information, a process we term
modality following. Prior work measured this behavior only with coarse
dataset-level statistics, overlooking the influence of model's confidence in
unimodal reasoning. In this paper, we introduce a new framework that decomposes
modality following into two fundamental factors: relative reasoning uncertainty
(the case-specific confidence gap between unimodal predictions) and inherent
modality preference( a model's stable bias when uncertainties are balanced). To
validate this framework, we construct a controllable dataset that
systematically varies the reasoning difficulty of visual and textual inputs.
Using entropy as a fine-grained uncertainty metric, we uncover a universal law:
the probability of following a modality decreases monotonically as its relative
uncertainty increases. At the relative difficulty level where the model tends
to follow both modalities with comparable probability what we call the balance
point, a practical indicator of the model's inherent preference. Unlike
traditional macro-level ratios, this measure offers a more principled and less
confounded way to characterize modality bias, disentangling it from unimodal
capabilities and dataset artifacts. Further, by probing layer-wise predictions,
we reveal the internal mechanism of oscillation: in ambiguous regions near the
balance point, models vacillate between modalities across layers, explaining
externally observed indecision. Together, these findings establish relative
uncertainty and inherent preference as the two governing principles of modality
following, offering both a quantitative framework and mechanistic insight into
how MLLMs resolve conflicting information.
690ae419d70e173c8452906b
Zhuoran Zhang
690ae419d70e173c8452906c
Tengyue Wang
690ae419d70e173c8452906d
Xilin Gong
690ae419d70e173c8452906e
Yang Shi
claimed_verified
2025-11-05T16:24:28.669Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
DogNeverSleep
user
DogNeverSleep
690ae419d70e173c8452906f
Haotian Wang
690ae419d70e173c84529070
Di Wang
690ae419d70e173c84529071
Lijie Hu
When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs
  Preference Dynamics in MLLMs
Multimodal large language models (MLLMs) must resolve conflicts when
different modalities provide contradictory information, a process we term
modality following. Prior work measured this behavior only with coarse
dataset-level statistics, overlooking the influence of model's confidence in
unimodal reasoning. In this paper, we introduce a new framework that decomposes
modality following into two fundamental factors: relative reasoning uncertainty
(the case-specific confidence gap between unimodal predictions) and inherent
modality preference( a model's stable bias when uncertainties are balanced). To
validate this framework, we construct a controllable dataset that
systematically varies the reasoning difficulty of visual and textual inputs.
Using entropy as a fine-grained uncertainty metric, we uncover a universal law:
the probability of following a modality decreases monotonically as its relative
uncertainty increases. At the relative difficulty level where the model tends
to follow both modalities with comparable probability what we call the balance
point, a practical indicator of the model's inherent preference. Unlike
traditional macro-level ratios, this measure offers a more principled and less
confounded way to characterize modality bias, disentangling it from unimodal
capabilities and dataset artifacts. Further, by probing layer-wise predictions,
we reveal the internal mechanism of oscillation: in ambiguous regions near the
balance point, models vacillate between modalities across layers, explaining
externally observed indecision. Together, these findings establish relative
uncertainty and inherent preference as the two governing principles of modality
following, offering both a quantitative framework and mechanistic insight into
how MLLMs resolve conflicting information.
A framework decomposes modality following in multimodal large language models into relative reasoning uncertainty and inherent modality preference, providing insights into how models resolve conflicting information.
multimodal large language models modality following relative reasoning uncertainty inherent modality preference entropy balance point layer-wise predictions