ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models
A recent trend in vision-language models (VLMs) has been to enhance their spatial cognition for embodied domains. Despite progress, existing evaluations have been limited both in paradigm and in coverage, hindering rapid, iterative model development. To address these limitations, we propose ESPIRE, a diagnostic benchmark for embodied spatial reasoning. ESPIRE offers a simulated world that physically grounds VLMs and evaluates them on spatial-reasoning-centric robotic tasks, thus narrowing the gap between evaluation and real-world deployment. To adapt VLMs to robotic tasks, we decompose each task into localization and execution, and frame both as generative problems, in stark contrast to predominant discriminative evaluations (e.g., via visual-question answering) that rely on distractors and discard execution. This decomposition further enables a fine-grained analysis beyond passive spatial reasoning toward reasoning to act. We systematically design ESPIRE both at the instruction level and at the environment level, ensuring broad coverage of spatial reasoning scenarios. We use ESPIRE to diagnose a range of frontier VLMs and provide in-depth analysis of their spatial reasoning behaviors.
69b792ac91a28162b233fca0
Yanpeng Zhao
admin_assigned
2026-03-16T18:13:19.082Z
6254b3add21e4cc386b71278
/avatars/1e739cc9d55a02e44cda2ce71f2d9c89.svg
Yanpeng Zhao
surprisal
user
surprisal
69b792ac91a28162b233fca1
Wentao Ding
claimed_verified
2026-03-19T13:03:48.306Z
6687b827840ee76959b8b373
/avatars/5dd91550679752868ece15c01b7f0faf.svg
WD
WD0X80
user
WD0X80
69b792ac91a28162b233fca2
Hongtao Li
claimed_verified
2026-03-16T10:39:38.324Z
64c20c59e75fd66a71d2b419
/avatars/7171fd26734672a4291595c12060d5df.svg
Hongtao Li
Aurumting
user
Aurumting
69b792ac91a28162b233fca3
Baoxiong Jia
69b792ac91a28162b233fca4
Zilong Zheng
ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models
A recent trend in vision-language models (VLMs) has been to enhance their spatial cognition for embodied domains. Despite progress, existing evaluations have been limited both in paradigm and in coverage, hindering rapid, iterative model development. To address these limitations, we propose ESPIRE, a diagnostic benchmark for embodied spatial reasoning. ESPIRE offers a simulated world that physically grounds VLMs and evaluates them on spatial-reasoning-centric robotic tasks, thus narrowing the gap between evaluation and real-world deployment. To adapt VLMs to robotic tasks, we decompose each task into localization and execution, and frame both as generative problems, in stark contrast to predominant discriminative evaluations (e.g., via visual-question answering) that rely on distractors and discard execution. This decomposition further enables a fine-grained analysis beyond passive spatial reasoning toward reasoning to act. We systematically design ESPIRE both at the instruction level and at the environment level, ensuring broad coverage of spatial reasoning scenarios. We use ESPIRE to diagnose a range of frontier VLMs and provide in-depth analysis of their spatial reasoning behaviors.
ESPIRE is a diagnostic benchmark for embodied spatial reasoning that evaluates vision-language models on robotic tasks through a decomposed localization and execution framework, enabling fine-grained analysis beyond passive spatial reasoning.
vision-language models embodied spatial reasoning diagnostic benchmark robotic tasks localization execution generative problems discriminative evaluations visual-question answering fine-grained analysis