More Images, More Problems? A Controlled Analysis of VLM Failure Modes
Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored. While existing benchmarks have initiated the evaluation of multi-image models, a comprehensive analysis of their core weaknesses and their causes is still lacking. In this work, we introduce MIMIC (Multi-Image Model Insights and Challenges), a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs. Using MIMIC, we conduct a series of diagnostic experiments that reveal pervasive issues: LVLMs often fail to aggregate information across images and struggle to track or attend to multiple concepts simultaneously. To address these failures, we propose two novel complementary remedies. On the data side, we present a procedural data-generation strategy that composes single-image annotations into rich, targeted multi-image training examples. On the optimization side, we analyze layer-wise attention patterns and derive an attention-masking scheme tailored for multi-image inputs. Experiments substantially improved cross-image aggregation, while also enhancing performance on existing multi-image benchmarks, outperforming prior state of the art across tasks. Data and code will be made available at https://github.com/anurag-198/MIMIC.
696df8ea3f1837bfb8970b04
Anurag Das
696df8ea3f1837bfb8970b05
Adrian Bulat
admin_assigned
2026-01-19T14:54:52.098Z
64a6e0923987f4dd3c37087d
/avatars/88bc86ddce1b38a012b07c81f5c61183.svg
Adrian Bulat
adrianb1
user
adrianb1
696df8ea3f1837bfb8970b06
Alberto Baldrati
claimed_verified
2026-01-27T09:44:06.108Z
61eeb1121e886efb9a8f57e3
https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg
Alberto Baldrati
ABaldrati
user
ABaldrati
696df8ea3f1837bfb8970b07
Ioannis Maniadis Metaxas
admin_assigned
2026-01-19T14:54:34.134Z
6731f7dbd2ef7f48198cdcd7
/avatars/a10ddf5e047eb0d8f480478d9fba30a1.svg
Ioannis Maniadis Metaxas
GManiadis
user
GManiadis
696df8ea3f1837bfb8970b08
Bernt Schiele
696df8ea3f1837bfb8970b09
Georgios Tzimiropoulos
admin_assigned
2026-01-19T14:55:01.071Z
649db0dcd8f9b2e2cb47e4d7
/avatars/e1fb64d5302271a3557ba306a85f2363.svg
Georgios Tzimiropoulos
tzimiro
user
tzimiro
696df8ea3f1837bfb8970b0a
Brais Martinez
More Images, More Problems? A Controlled Analysis of VLM Failure Modes
Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored. While existing benchmarks have initiated the evaluation of multi-image models, a comprehensive analysis of their core weaknesses and their causes is still lacking. In this work, we introduce MIMIC (Multi-Image Model Insights and Challenges), a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs. Using MIMIC, we conduct a series of diagnostic experiments that reveal pervasive issues: LVLMs often fail to aggregate information across images and struggle to track or attend to multiple concepts simultaneously. To address these failures, we propose two novel complementary remedies. On the data side, we present a procedural data-generation strategy that composes single-image annotations into rich, targeted multi-image training examples. On the optimization side, we analyze layer-wise attention patterns and derive an attention-masking scheme tailored for multi-image inputs. Experiments substantially improved cross-image aggregation, while also enhancing performance on existing multi-image benchmarks, outperforming prior state of the art across tasks. Data and code will be made available at https://github.com/anurag-198/MIMIC.
Large Vision Language Models exhibit significant limitations in multi-image understanding and reasoning, which are revealed through a new benchmark and addressed via procedural data generation and attention masking techniques.
Large Vision Language Models multi-image capabilities benchmark diagnostic experiments cross-image aggregation attention-masking scheme procedural data-generation strategy