GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
Visual grounding, localizing objects from natural language descriptions, represents a critical bridge between language and vision understanding. While multimodal large language models (MLLMs) achieve impressive scores on existing benchmarks, a fundamental question remains: can MLLMs truly ground language in vision with human-like sophistication, or are they merely pattern-matching on simplified datasets? Current benchmarks fail to capture real-world complexity where humans effortlessly navigate ambiguous references and recognize when grounding is impossible. To rigorously assess MLLMs' true capabilities, we introduce GroundingME, a benchmark that systematically challenges models across four critical dimensions: (1) Discriminative, distinguishing highly similar objects, (2) Spatial, understanding complex relational descriptions, (3) Limited, handling occlusions or tiny objects, and (4) Rejection, recognizing ungroundable queries. Through careful curation combining automated generation with human verification, we create 1,005 challenging examples mirroring real-world complexity. Evaluating 25 state-of-the-art MLLMs reveals a profound capability gap: the best model achieves only 45.1% accuracy, while most score 0% on rejection tasks, reflexively hallucinating objects rather than acknowledging their absence, raising critical safety concerns for deployment. We explore two strategies for improvements: (1) test-time scaling selects optimal response by thinking trajectory to improve complex grounding by up to 2.9%, and (2) data-mixture training teaches models to recognize ungroundable queries, boosting rejection accuracy from 0% to 27.9%. GroundingME thus serves as both a diagnostic tool revealing current limitations in MLLMs and a roadmap toward human-level visual grounding.
6948dc2634f46eaf46cbb34d
Rang Li
claimed_verified
2025-12-25T20:51:46.271Z
6847abd1892b779986a5ec80
/avatars/0f820eedf74d8e0636af94ef7cd4d04c.svg
Rang Li
lirang04
user
lirang04
6948dc2634f46eaf46cbb34e
Lei Li
claimed_verified
2025-12-22T10:58:23.401Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
tobiaslee
user
tobiaslee
6948dc2634f46eaf46cbb34f
Shuhuai Ren
6948dc2634f46eaf46cbb350
Hao Tian
6948dc2634f46eaf46cbb351
Shuhao Gu
6948dc2634f46eaf46cbb352
Shicheng Li
6948dc2634f46eaf46cbb353
Zihao Yue
6948dc2634f46eaf46cbb354
Yudong Wang
6948dc2634f46eaf46cbb355
Wenhan Ma
6948dc2634f46eaf46cbb356
Zhe Yang
6948dc2634f46eaf46cbb357
Jingyuan Ma
6948dc2634f46eaf46cbb358
Zhifang Sui
6948dc2634f46eaf46cbb359
Fuli Luo
GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
Visual grounding, localizing objects from natural language descriptions, represents a critical bridge between language and vision understanding. While multimodal large language models (MLLMs) achieve impressive scores on existing benchmarks, a fundamental question remains: can MLLMs truly ground language in vision with human-like sophistication, or are they merely pattern-matching on simplified datasets? Current benchmarks fail to capture real-world complexity where humans effortlessly navigate ambiguous references and recognize when grounding is impossible. To rigorously assess MLLMs' true capabilities, we introduce GroundingME, a benchmark that systematically challenges models across four critical dimensions: (1) Discriminative, distinguishing highly similar objects, (2) Spatial, understanding complex relational descriptions, (3) Limited, handling occlusions or tiny objects, and (4) Rejection, recognizing ungroundable queries. Through careful curation combining automated generation with human verification, we create 1,005 challenging examples mirroring real-world complexity. Evaluating 25 state-of-the-art MLLMs reveals a profound capability gap: the best model achieves only 45.1% accuracy, while most score 0% on rejection tasks, reflexively hallucinating objects rather than acknowledging their absence, raising critical safety concerns for deployment. We explore two strategies for improvements: (1) test-time scaling selects optimal response by thinking trajectory to improve complex grounding by up to 2.9%, and (2) data-mixture training teaches models to recognize ungroundable queries, boosting rejection accuracy from 0% to 27.9%. GroundingME thus serves as both a diagnostic tool revealing current limitations in MLLMs and a roadmap toward human-level visual grounding.
GroundingME benchmark evaluates multimodal large language models' visual grounding capabilities across complexity dimensions, revealing significant gaps and proposing strategies for improvement.
multimodal large language models visual grounding GroundingME discriminative spatial limited rejection thinking trajectory data-mixture training