Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models
Vision-language models (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.
694b766b746a34b55dd53de6
Shengchao Zhou
claimed_verified
2025-12-25T20:45:38.300Z
658d3b74f893598fcaee75f1
/avatars/edb2243ad020bd72a1b305accc2e7034.svg
Shengchao Zhou
zhousc
user
zhousc
694b766b746a34b55dd53de7
Yuxin Chen
claimed_verified
2025-12-25T20:45:35.585Z
669f3b098c65c172c4d64039
/avatars/d85158964853ab87b9b677fa16df90f8.svg
Yuxin Chen
Uasonchen
user
Uasonchen
694b766b746a34b55dd53de8
Yuying Ge
694b766b746a34b55dd53de9
Wei Huang
claimed_verified
2025-12-25T20:45:32.801Z
656db3f53dc1d277e5a64410
https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
Wei Huang
AaronHuangWei
user
AaronHuangWei
694b766b746a34b55dd53dea
Jiehong Lin
694b766b746a34b55dd53deb
Ying Shan
694b766b746a34b55dd53dec
Xiaojuan Qi
Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models
Vision-language models (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.
DSR Suite enhances vision-language models with dynamic spatial reasoning through automated data generation and a geometry selection module that integrates geometric priors.
vision-language models dynamic spatial reasoning 4D-aware training automated pipeline multiple-choice question-answer pairs vision foundation models camera poses local point clouds object masks orientations 3D trajectories DSR-Train DSR-Bench Geometry Selection Module geometry tokens Qwen2.5-VL-7B video understanding benchmarks