MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-world applications. However, existing robustness benchmarks typically focus on hallucination or misleading textual inputs, while largely overlooking the equally critical challenge posed by misleading visual inputs in assessing visual understanding. To fill this important gap, we introduce MVI-Bench, the first comprehensive benchmark specially designed for evaluating how Misleading Visual Inputs undermine the robustness of LVLMs. Grounded in fundamental visual primitives, the design of MVI-Bench centers on three hierarchical levels of misleading visual inputs: Visual Concept, Visual Attribute, and Visual Relationship. Using this taxonomy, we curate six representative categories and compile 1,248 expertly annotated VQA instances. To facilitate fine-grained robustness evaluation, we further introduce MVI-Sensitivity, a novel metric that characterizes LVLM robustness at a granular level. Empirical results across 18 state-of-the-art LVLMs uncover pronounced vulnerabilities to misleading visual inputs, and our in-depth analyses on MVI-Bench provide actionable insights that can guide the development of more reliable and robust LVLMs. The benchmark and codebase can be accessed at https://github.com/chenyil6/MVI-Bench.
691d34810247f3d258ef322b
Huiyi Chen
691d34810247f3d258ef322c
Jiawei Peng
691d34810247f3d258ef322d
Dehai Min
claimed_verified
2025-11-27T10:00:42.886Z
629c6ee73a3221bb210afc2d
https://cdn-avatars.huggingface.co/v1/production/uploads/629c6ee73a3221bb210afc2d/Mg-VymVvHQn_pDrTgks0s.jpeg
Dehai Min
ZhishanQ
user
ZhishanQ
691d34810247f3d258ef322e
Changchang Sun
691d34810247f3d258ef322f
Kaijie Chen
691d34810247f3d258ef3230
Yan Yan
691d34810247f3d258ef3231
Xu Yang
691d34810247f3d258ef3232
Lu Cheng
MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-world applications. However, existing robustness benchmarks typically focus on hallucination or misleading textual inputs, while largely overlooking the equally critical challenge posed by misleading visual inputs in assessing visual understanding. To fill this important gap, we introduce MVI-Bench, the first comprehensive benchmark specially designed for evaluating how Misleading Visual Inputs undermine the robustness of LVLMs. Grounded in fundamental visual primitives, the design of MVI-Bench centers on three hierarchical levels of misleading visual inputs: Visual Concept, Visual Attribute, and Visual Relationship. Using this taxonomy, we curate six representative categories and compile 1,248 expertly annotated VQA instances. To facilitate fine-grained robustness evaluation, we further introduce MVI-Sensitivity, a novel metric that characterizes LVLM robustness at a granular level. Empirical results across 18 state-of-the-art LVLMs uncover pronounced vulnerabilities to misleading visual inputs, and our in-depth analyses on MVI-Bench provide actionable insights that can guide the development of more reliable and robust LVLMs. The benchmark and codebase can be accessed at https://github.com/chenyil6/MVI-Bench.
MVI-Bench evaluates the robustness of Large Vision-Language Models against misleading visual inputs using a hierarchical taxonomy and a novel sensitivity metric, revealing significant vulnerabilities.
Large Vision-Language Models LVLMs robustness hallucination misleading visual inputs visual primitives Visual Concept Visual Attribute Visual Relationship VQA MVI-Bench MVI-Sensitivity