Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
We introduce two new benchmarks REST and REST+(Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.
69392639dfc35938ba12a179
Angela van Sprang
admin_assigned
2025-12-10T15:49:24.707Z
661797a78d7b07669c896092
/avatars/971845551815d7826f97100f98df713b.svg
Angela van Sprang
Angela0603
user
Angela0603
69392639dfc35938ba12a17a
Laurens Samson
69392639dfc35938ba12a17b
Ana Lucic
69392639dfc35938ba12a17c
Erman Acar
admin_assigned
2025-12-10T15:49:55.886Z
67adeeeff7d84e72910cafd2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/X9vk10KOuOrWuaEMBCoWz.png
Erman Acar
ermancr
user
ermancr
69392639dfc35938ba12a17d
Sennay Ghebreab
69392639dfc35938ba12a17e
Yuki M. Asano
admin_assigned
2025-12-10T15:49:46.462Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
yukimasano
user
yukimasano
Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
We introduce two new benchmarks REST and REST+(Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.
Two new benchmarks assess cross-modal inconsistency in multimodal large language models, showing significant variability and impact of visual characteristics on performance.
multimodal large language models embedding space cross-modal inconsistency REST REST+ Render-Equivalence Stress Tests OCR vision tokens modality gap