SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
Despite impressive high-level video comprehension, multimodal language models
struggle with spatial reasoning across time and space. While current spatial
training approaches rely on real-world video data, obtaining diverse footage
with precise spatial annotations remains a bottleneck. To alleviate this
bottleneck, we present SIMS-V -- a systematic data-generation framework that
leverages the privileged information of 3D simulators to create spatially-rich
video training data for multimodal language models. Using this framework, we
investigate which properties of simulated data drive effective real-world
transfer through systematic ablations of question types, mixes, and scales. We
identify a minimal set of three question categories (metric measurement,
perspective-dependent reasoning, and temporal tracking) that prove most
effective for developing transferable spatial intelligence, outperforming
comprehensive coverage despite using fewer question types. These insights
enable highly efficient training: our 7B-parameter video LLM fine-tuned on just
25K simulated examples outperforms the larger 72B baseline and achieves
competitive performance with proprietary models on rigorous real-world spatial
reasoning benchmarks. Our approach demonstrates robust generalization,
maintaining performance on general video understanding while showing
substantial improvements on embodied and real-world spatial tasks.
690df354ad2597bf6c465057
Ellis Brown
claimed_verified
2025-11-10T09:30:47.346Z
626dc5105f7327906f0b2a4e
https://cdn-avatars.huggingface.co/v1/production/uploads/626dc5105f7327906f0b2a4e/QCSzuwYqsv8ozRnusVb-F.jpeg
Ellis Brown
ellisbrown
user
ellisbrown
690df354ad2597bf6c465058
Arijit Ray
claimed_verified
2025-11-10T09:30:45.236Z
6303f2f01dd5d3c6248559ae
https://cdn-avatars.huggingface.co/v1/production/uploads/6303f2f01dd5d3c6248559ae/1l18-bKuTcWSRZdukBFDm.jpeg
Arijit
array
user
array
690df354ad2597bf6c465059
Ranjay Krishna
690df354ad2597bf6c46505a
Ross Girshick
690df354ad2597bf6c46505b
Rob Fergus
690df354ad2597bf6c46505c
Saining Xie
claimed_verified
2025-11-11T19:51:30.213Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
sainx
user
sainx
SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
Despite impressive high-level video comprehension, multimodal language models
struggle with spatial reasoning across time and space. While current spatial
training approaches rely on real-world video data, obtaining diverse footage
with precise spatial annotations remains a bottleneck. To alleviate this
bottleneck, we present SIMS-V -- a systematic data-generation framework that
leverages the privileged information of 3D simulators to create spatially-rich
video training data for multimodal language models. Using this framework, we
investigate which properties of simulated data drive effective real-world
transfer through systematic ablations of question types, mixes, and scales. We
identify a minimal set of three question categories (metric measurement,
perspective-dependent reasoning, and temporal tracking) that prove most
effective for developing transferable spatial intelligence, outperforming
comprehensive coverage despite using fewer question types. These insights
enable highly efficient training: our 7B-parameter video LLM fine-tuned on just
25K simulated examples outperforms the larger 72B baseline and achieves
competitive performance with proprietary models on rigorous real-world spatial
reasoning benchmarks. Our approach demonstrates robust generalization,
maintaining performance on general video understanding while showing
substantial improvements on embodied and real-world spatial tasks.
A data-generation framework using 3D simulators improves spatial reasoning in multimodal language models with efficient training on simulated data.
multimodal language models spatial reasoning 3D simulators video training data systematic ablations question types metric measurement perspective-dependent reasoning temporal tracking video LLM fine-tuning real-world spatial reasoning benchmarks embodied tasks