FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos
Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent motion benchmarks, primarily due to the scarcity of large-scale, fine-grained motion datasets. Existing motion datasets are often constructed from costly manual annotation, severely limiting scalability. To address this challenge, we introduce FoundationMotion, a fully automated data curation pipeline that constructs large-scale motion datasets. Our approach first detects and tracks objects in videos to extract their trajectories, then leverages these trajectories and video frames with Large Language Models (LLMs) to generate fine-grained captions and diverse question-answer pairs about motion and spatial reasoning. Using datasets produced by this pipeline, we fine-tune open-source models including NVILA-Video-15B and Qwen2.5-7B, achieving substantial improvements in motion understanding without compromising performance on other tasks. Notably, our models outperform strong closed-source baselines like Gemini-2.5 Flash and large open-source models such as Qwen2.5-VL-72B across diverse motion understanding datasets and benchmarks. FoundationMotion thus provides a scalable solution for curating fine-grained motion datasets that enable effective fine-tuning of diverse models to enhance motion understanding and spatial reasoning capabilities.
69417bc965f1e24a117808db
Yulu Gan
69417bc965f1e24a117808dc
Ligeng Zhu
69417bc965f1e24a117808dd
Dandan Shan
69417bc965f1e24a117808de
Baifeng Shi
69417bc965f1e24a117808df
Hongxu Yin
69417bc965f1e24a117808e0
Boris Ivanovic
69417bc965f1e24a117808e1
Song Han
69417bc965f1e24a117808e2
Trevor Darrell
69417bc965f1e24a117808e3
Jitendra Malik
69417bc965f1e24a117808e4
Marco Pavone
69417bc965f1e24a117808e5
Boyi Li
claimed_verified
2025-12-17T10:09:03.650Z
620dd3888528f797e88cb9b5
/avatars/af04728788d78fe7d6375e19e32a535e.svg
Boyi Li
Boyiliee
user
Boyiliee
FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos
Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent motion benchmarks, primarily due to the scarcity of large-scale, fine-grained motion datasets. Existing motion datasets are often constructed from costly manual annotation, severely limiting scalability. To address this challenge, we introduce FoundationMotion, a fully automated data curation pipeline that constructs large-scale motion datasets. Our approach first detects and tracks objects in videos to extract their trajectories, then leverages these trajectories and video frames with Large Language Models (LLMs) to generate fine-grained captions and diverse question-answer pairs about motion and spatial reasoning. Using datasets produced by this pipeline, we fine-tune open-source models including NVILA-Video-15B and Qwen2.5-7B, achieving substantial improvements in motion understanding without compromising performance on other tasks. Notably, our models outperform strong closed-source baselines like Gemini-2.5 Flash and large open-source models such as Qwen2.5-VL-72B across diverse motion understanding datasets and benchmarks. FoundationMotion thus provides a scalable solution for curating fine-grained motion datasets that enable effective fine-tuning of diverse models to enhance motion understanding and spatial reasoning capabilities.
FoundationMotion is an automated pipeline for creating large-scale motion datasets using object detection, trajectory extraction, and LLM-generated captions, improving motion understanding in models.
motion understanding state-of-the-art models motion benchmarks motion datasets manual annotation data curation pipeline object detection trajectory extraction Large Language Models fine-grained captions question-answer pairs NVILA-Video-15B Qwen2.5-7B Gemini-2.5 Flash Qwen2.5-VL-72B spatial reasoning