Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the modality gap is task- and data-dependent. For example, math tasks degrade by over 60 points on synthetic renderings, while natural document images often match or exceed text-mode performance. Rendering choices such as font and resolution are strong confounds, with font alone swinging accuracy by up to 47 percentage points. To understand this, we conduct a grounded-theory error analysis of over 4,000 examples, revealing that image mode selectively amplifies reading errors (calculation and formatting failures) while leaving knowledge and reasoning errors largely unchanged, and that some models exhibit a chain-of-thought reasoning collapse under visual input. Motivated by these findings, we propose a self-distillation method that trains the model on its own pure text reasoning traces paired with image inputs, raising image-mode accuracy on GSM8K from 30.71% to 92.72% and transferring to unseen benchmarks without catastrophic forgetting. Overall, our study provides a systematic understanding of the modality gap and suggests a practical path toward improving visual text understanding in multimodal language models.
69b0c7481f7902d47f6df948
Kaiser Sun
claimed_verified
2026-03-11T08:44:16.493Z
616745f893f6837511d9ae25
https://cdn-avatars.huggingface.co/v1/production/uploads/616745f893f6837511d9ae25/H0JIGDL5VOu29zVd4KiKB.jpeg
Kaiser Sun
KaiserWhoLearns
user
KaiserWhoLearns
69b0c7481f7902d47f6df949
Xiaochuang Yuan
claimed_verified
2026-03-11T08:32:59.575Z
68354cb044d0526c43242048
https://cdn-avatars.huggingface.co/v1/production/uploads/68354cb044d0526c43242048/WxstE36Y4_Z8Aty3M2VMB.jpeg
Xiaochuang Yuan
yxc20098
user
yxc20098
69b0c7481f7902d47f6df94a
Hongjun Liu
claimed_verified
2026-03-11T08:33:02.873Z
646474126d621be0dd08d6d4
/avatars/5041573e603225e9de3b689efd1dd499.svg
Liu Hongjun
Jan150000
user
Jan150000
69b0c7481f7902d47f6df94b
Chen Zhao
69b0c7481f7902d47f6df94c
Cheng Zhang
69b0c7481f7902d47f6df94d
Mark Dredze
admin_assigned
2026-03-11T09:45:34.135Z
6140f23f6dc32b0812420fc5
https://cdn-avatars.huggingface.co/v1/production/uploads/1631646255954-noauth.jpeg
Mark Dredze
mdredze
user
mdredze
69b0c7481f7902d47f6df94e
Fan Bai
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the modality gap is task- and data-dependent. For example, math tasks degrade by over 60 points on synthetic renderings, while natural document images often match or exceed text-mode performance. Rendering choices such as font and resolution are strong confounds, with font alone swinging accuracy by up to 47 percentage points. To understand this, we conduct a grounded-theory error analysis of over 4,000 examples, revealing that image mode selectively amplifies reading errors (calculation and formatting failures) while leaving knowledge and reasoning errors largely unchanged, and that some models exhibit a chain-of-thought reasoning collapse under visual input. Motivated by these findings, we propose a self-distillation method that trains the model on its own pure text reasoning traces paired with image inputs, raising image-mode accuracy on GSM8K from 30.71% to 92.72% and transferring to unseen benchmarks without catastrophic forgetting. Overall, our study provides a systematic understanding of the modality gap and suggests a practical path toward improving visual text understanding in multimodal language models.
Multimodal large language models exhibit inconsistent performance when processing text from images versus textual tokens, with factors like rendering quality and task type influencing this modality gap, which can be mitigated through self-distillation techniques that leverage text-based reasoning traces.
multimodal large language models modality gap visual text understanding self-distillation reasoning traces GSM8K