MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence
Spatial understanding over continuous visual input is crucial for MLLMs to evolve into general-purpose assistants in physical environments. Yet there is still no comprehensive benchmark that holistically assesses the progress toward this goal. In this work, we introduce MMSI-Video-Bench, a fully human-annotated benchmark for video-based spatial intelligence in MLLMs. It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips from 25 datasets and in-house videos. Each item is carefully designed and reviewed by 3DV experts with explanatory rationales to ensure precise, unambiguous grounding. Leveraging its diverse data sources and holistic task coverage, MMSI-Video-Bench also supports three domain-oriented sub-benchmarks (Indoor Scene Perception Bench, Robot Bench and Grounding Bench) for targeted capability assessment. We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human--AI gap: many models perform near chance, and the best reasoning model lags humans by nearly 60%. We further find that spatially fine-tuned models still fail to generalize effectively on our benchmark. Fine-grained error analysis exposes systematic failures in geometric reasoning, motion grounding, long-horizon prediction, and cross-video correspondence. We also show that typical frame-sampling strategies transfer poorly to our reasoning-intensive benchmark, and that neither 3D spatial cues nor chain-of-thought prompting yields meaningful gains. We expect our benchmark to establish a solid testbed for advancing video-based spatial intelligence.
6942c775fb33037a39577dcc
Jingli Lin
claimed_verified
2025-12-18T08:00:14.115Z
666b0210b307f882e0724b7d
/avatars/80ad8ac0a4cec885158607fc634c1291.svg
Jingli Lin
rbler
user
rbler
6942c775fb33037a39577dcd
Runsen Xu
6942c775fb33037a39577dce
Shaohao Zhu
6942c775fb33037a39577dcf
Sihan Yang
6942c775fb33037a39577dd0
Peizhou Cao
6942c775fb33037a39577dd1
Yunlong Ran
6942c775fb33037a39577dd2
Miao Hu
6942c775fb33037a39577dd3
Chenming Zhu
6942c775fb33037a39577dd4
Yiman Xie
6942c775fb33037a39577dd5
Yilin Long
6942c775fb33037a39577dd6
Wenbo Hu
claimed_verified
2026-04-10T07:45:39.909Z
645940c8f1f5263c2ef334ab
/avatars/e908686d0e1b2708adb0e30e596a6f39.svg
Wenbo Hu
gordonhu
user
gordonhu
6942c775fb33037a39577dd7
Dahua Lin
6942c775fb33037a39577dd8
Tai Wang
6942c775fb33037a39577dd9
Jiangmiao Pang
MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence
Spatial understanding over continuous visual input is crucial for MLLMs to evolve into general-purpose assistants in physical environments. Yet there is still no comprehensive benchmark that holistically assesses the progress toward this goal. In this work, we introduce MMSI-Video-Bench, a fully human-annotated benchmark for video-based spatial intelligence in MLLMs. It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips from 25 datasets and in-house videos. Each item is carefully designed and reviewed by 3DV experts with explanatory rationales to ensure precise, unambiguous grounding. Leveraging its diverse data sources and holistic task coverage, MMSI-Video-Bench also supports three domain-oriented sub-benchmarks (Indoor Scene Perception Bench, Robot Bench and Grounding Bench) for targeted capability assessment. We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human--AI gap: many models perform near chance, and the best reasoning model lags humans by nearly 60%. We further find that spatially fine-tuned models still fail to generalize effectively on our benchmark. Fine-grained error analysis exposes systematic failures in geometric reasoning, motion grounding, long-horizon prediction, and cross-video correspondence. We also show that typical frame-sampling strategies transfer poorly to our reasoning-intensive benchmark, and that neither 3D spatial cues nor chain-of-thought prompting yields meaningful gains. We expect our benchmark to establish a solid testbed for advancing video-based spatial intelligence.
MMSI-Video-Bench is a comprehensive benchmark for video-based spatial intelligence in MLLMs, revealing significant gaps between human and AI performance and highlighting challenges in geometric reasoning, motion grounding, and cross-video correspondence.
MMSI-Video-Bench spatial intelligence MLLMs Perception Planning Prediction Cross-Video Reasoning 3DV experts fine-grained error analysis geometric reasoning motion grounding long-horizon prediction cross-video correspondence frame-sampling strategies 3D spatial cues chain-of-thought prompting