Robot Learning from a Physical World Model
We introduce PhysWorld, a framework that enables robot learning from video
generation through physical world modeling. Recent video generation models can
synthesize photorealistic visual demonstrations from language commands and
images, offering a powerful yet underexplored source of training signals for
robotics. However, directly retargeting pixel motions from generated videos to
robots neglects physics, often resulting in inaccurate manipulations. PhysWorld
addresses this limitation by coupling video generation with physical world
reconstruction. Given a single image and a task command, our method generates
task-conditioned videos and reconstructs the underlying physical world from the
videos, and the generated video motions are grounded into physically accurate
actions through object-centric residual reinforcement learning with the
physical world model. This synergy transforms implicit visual guidance into
physically executable robotic trajectories, eliminating the need for real robot
data collection and enabling zero-shot generalizable robotic manipulation.
Experiments on diverse real-world tasks demonstrate that PhysWorld
substantially improves manipulation accuracy compared to previous approaches.
Visit https://pointscoder.github.io/PhysWorld_Web/{the project webpage}
for details.
6912b819a644ba07c499c765
Jiageng Mao
6912b819a644ba07c499c766
Sicheng He
6912b819a644ba07c499c767
Hao-Ning Wu
6912b819a644ba07c499c768
Yang You
6912b819a644ba07c499c769
Shuyang Sun
6912b819a644ba07c499c76a
Zhicheng Wang
6912b819a644ba07c499c76b
Yanan Bao
6912b819a644ba07c499c76c
Huizhong Chen
6912b819a644ba07c499c76d
Leonidas Guibas
6912b819a644ba07c499c76e
Vitor Guizilini
6912b819a644ba07c499c76f
Howard Zhou
6912b819a644ba07c499c770
Yue Wang
Robot Learning from a Physical World Model
We introduce PhysWorld, a framework that enables robot learning from video
generation through physical world modeling. Recent video generation models can
synthesize photorealistic visual demonstrations from language commands and
images, offering a powerful yet underexplored source of training signals for
robotics. However, directly retargeting pixel motions from generated videos to
robots neglects physics, often resulting in inaccurate manipulations. PhysWorld
addresses this limitation by coupling video generation with physical world
reconstruction. Given a single image and a task command, our method generates
task-conditioned videos and reconstructs the underlying physical world from the
videos, and the generated video motions are grounded into physically accurate
actions through object-centric residual reinforcement learning with the
physical world model. This synergy transforms implicit visual guidance into
physically executable robotic trajectories, eliminating the need for real robot
data collection and enabling zero-shot generalizable robotic manipulation.
Experiments on diverse real-world tasks demonstrate that PhysWorld
substantially improves manipulation accuracy compared to previous approaches.
Visit https://pointscoder.github.io/PhysWorld_Web/{the project webpage}
for details.
PhysWorld integrates video generation and physical world modeling to enable accurate robotic manipulation from visual demonstrations without real robot data.
video generation physical world modeling task-conditioned videos object-centric residual reinforcement learning physically executable robotic trajectories zero-shot generalizable robotic manipulation