DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.
69894c74beecc443208d25db
Shenyuan Gao
69894c74beecc443208d25dc
William Liang
69894c74beecc443208d25dd
Kaiyuan Zheng
69894c74beecc443208d25de
Ayaan Malik
69894c74beecc443208d25df
Seonghyeon Ye
69894c74beecc443208d25e0
Sihyun Yu
69894c74beecc443208d25e1
Wei-Cheng Tseng
69894c74beecc443208d25e2
Yuzhu Dong
69894c74beecc443208d25e3
Kaichun Mo
69894c74beecc443208d25e4
Chen-Hsuan Lin
69894c74beecc443208d25e5
Qianli Ma
69894c74beecc443208d25e6
Seungjun Nah
69894c74beecc443208d25e7
Loic Magne
69894c74beecc443208d25e8
Jiannan Xiang
69894c74beecc443208d25e9
Yuqi Xie
69894c74beecc443208d25ea
Ruijie Zheng
69894c74beecc443208d25eb
Dantong Niu
69894c74beecc443208d25ec
You Liang Tan
69894c74beecc443208d25ed
K. R. Zentner
69894c74beecc443208d25ee
George Kurian
69894c74beecc443208d25ef
Suneel Indupuru
69894c74beecc443208d25f0
Pooya Jannaty
69894c74beecc443208d25f1
Jinwei Gu
69894c74beecc443208d25f2
Jun Zhang
69894c74beecc443208d25f3
Jitendra Malik
69894c74beecc443208d25f4
Pieter Abbeel
69894c74beecc443208d25f5
Ming-Yu Liu
69894c74beecc443208d25f6
Yuke Zhu
69894c74beecc443208d25f7
Joel Jang
69894c74beecc443208d25f8
Linxi "Jim" Fan
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.
DreamDojo is a foundation world model trained on 44k hours of egocentric human videos that enables efficient simulation of dexterous robotic tasks through continuous latent actions and real-time distillation.
world model egocentric videos continuous latent actions action labels distillation pipeline real-time speed teleoperation policy evaluation model-based planning out-of-distribution benchmarks