VideoWorld 2: Learning Transferable Knowledge from Real-world Videos
Learning transferable knowledge from unlabeled video data and applying it in new environments is a fundamental capability of intelligent agents. This work presents VideoWorld 2, which extends VideoWorld and offers the first investigation into learning transferable knowledge directly from raw real-world videos. At its core, VideoWorld 2 introduces a dynamic-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance: a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics. These latent codes are then modeled autoregressively to learn task policies and support long-horizon reasoning. We evaluate VideoWorld 2 on challenging real-world handcraft making tasks, where prior video generation and latent-dynamics models struggle to operate reliably. Remarkably, VideoWorld 2 achieves up to 70% improvement in task success rate and produces coherent long execution videos. In robotics, we show that VideoWorld 2 can acquire effective manipulation knowledge from the Open-X dataset, which substantially improves task performance on CALVIN. This study reveals the potential of learning transferable world knowledge directly from raw videos, with all code, data, and models to be open-sourced for further research.
698bf2046052d3bed9630a4d
Zhongwei Ren
698bf2046052d3bed9630a4e
Yunchao Wei
698bf2046052d3bed9630a4f
Xiao Yu
698bf2046052d3bed9630a50
Guixun Luo
698bf2046052d3bed9630a51
Yao Zhao
698bf2046052d3bed9630a52
Bingyi Kang
698bf2046052d3bed9630a53
Jiashi Feng
698bf2046052d3bed9630a54
Xiaojie Jin
VideoWorld 2: Learning Transferable Knowledge from Real-world Videos
Learning transferable knowledge from unlabeled video data and applying it in new environments is a fundamental capability of intelligent agents. This work presents VideoWorld 2, which extends VideoWorld and offers the first investigation into learning transferable knowledge directly from raw real-world videos. At its core, VideoWorld 2 introduces a dynamic-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance: a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics. These latent codes are then modeled autoregressively to learn task policies and support long-horizon reasoning. We evaluate VideoWorld 2 on challenging real-world handcraft making tasks, where prior video generation and latent-dynamics models struggle to operate reliably. Remarkably, VideoWorld 2 achieves up to 70% improvement in task success rate and produces coherent long execution videos. In robotics, we show that VideoWorld 2 can acquire effective manipulation knowledge from the Open-X dataset, which substantially improves task performance on CALVIN. This study reveals the potential of learning transferable world knowledge directly from raw videos, with all code, data, and models to be open-sourced for further research.
VideoWorld 2 enables transferable knowledge learning from raw videos through a dynamic-enhanced Latent Dynamics Model that decouples action dynamics from visual appearance, achieving improved task performance and long-horizon reasoning in real-world applications.
latent dynamics model video diffusion model action dynamics visual appearance latent codes autoregressive modeling task policies long-horizon reasoning Open-X dataset CALVIN