DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.
693251cd6d1060ca587a2731
Kairun Wen
claimed_verified
2025-12-05T15:14:53.159Z
646a31bd3eb2bab0419a54ef
https://cdn-avatars.huggingface.co/v1/production/uploads/646a31bd3eb2bab0419a54ef/RD2GsZdewSZz7VUwWabUV.png
Kairun Wen
kairunwen
user
kairunwen
693251cd6d1060ca587a2732
Yuzhi Huang
693251cd6d1060ca587a2733
Runyu Chen
693251cd6d1060ca587a2734
Hui Zheng
693251cd6d1060ca587a2735
Yunlong Lin
claimed_verified
2025-12-16T09:48:59.489Z
64ecb174f22081b4ac7ca397
https://cdn-avatars.huggingface.co/v1/production/uploads/64ecb174f22081b4ac7ca397/PiAPtD_rbuhGOqfE6ZSIu.jpeg
Yunlong Lin
LYL1015
user
LYL1015
693251cd6d1060ca587a2736
Panwang Pan
claimed_verified
2025-12-10T09:10:14.494Z
654866e8cd0a5621395f8287
https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
Panwang Pan
paulpanwang
user
paulpanwang
693251cd6d1060ca587a2737
Chenxin Li
693251cd6d1060ca587a2738
Wenyan Cong
693251cd6d1060ca587a2739
Jian Zhang
693251cd6d1060ca587a273a
Junbin Lu
693251cd6d1060ca587a273b
Chenguo Lin
claimed_verified
2025-12-12T09:17:55.077Z
62e18206926f4892a4c782bd
https://cdn-avatars.huggingface.co/v1/production/uploads/62e18206926f4892a4c782bd/m_OAeRb2a7a5PlnK6d4Dg.jpeg
Chenguo Lin
chenguolin
user
chenguolin
693251cd6d1060ca587a273c
Dilin Wang
693251cd6d1060ca587a273d
Zhicheng Yan
693251cd6d1060ca587a273e
Hongyu Xu
693251cd6d1060ca587a273f
Justin Theiss
693251cd6d1060ca587a2740
Yue Huang
693251cd6d1060ca587a2741
Xinghao Ding
693251cd6d1060ca587a2742
Rakesh Ranjan
693251cd6d1060ca587a2743
Zhiwen Fan
DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.
DynamicVerse is a framework that models dynamic real-world videos by integrating large vision, geometric, and multimodal models to produce a comprehensive 4D multimodal dataset, achieving superior performance in video depth estimation, camera pose estimation, and camera intrinsics estimation.
DynamicVerse multimodal 4D world modeling Bundle Adjustment global optimization video depth estimation camera pose estimation camera intrinsics estimation