Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models
Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap Digital Agnosia. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.
69e02f195b704445ba50b532
Yunkai Zhang
69e02f195b704445ba50b533
Linda Li
69e02f195b704445ba50b534
Yingxin Cui
69e02f195b704445ba50b535
Xiyuan Ruan
claimed_verified
2026-04-16T08:02:12.662Z
67353507847400c04ad98ba9
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/KlbKkv3EYPZN4PnLo_eLO.png
Raina Ruan
Rrbabe
user
Rrbabe
69e02f195b704445ba50b536
Zeyu Zheng
69e02f195b704445ba50b537
Kezhen Chen
69e02f195b704445ba50b538
Yi Zhang
69e02f195b704445ba50b539
Diji Yang
claimed_verified
2026-04-16T08:02:14.927Z
62a17ca6d57bf9ab9c697206
/avatars/40cca602d481382b0adb13996328b515.svg
Diji Yang
dyang39
user
dyang39
Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models
Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap Digital Agnosia. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.
Vision-language models fail to accurately reproduce visual details in controlled grid-to-matrix tasks, revealing a disconnect between visual encoding and language output that persists despite model scaling and alignment improvements.
Vision-Language Models visual encoders end-to-end evaluation Digital Agnosia visual patch boundaries multimodal alignment