Toward Physically Consistent Driving Video World Models under Challenging Trajectories
Video generation models have shown strong potential as world models for autonomous driving simulation. However, existing approaches are primarily trained on real-world driving datasets, which mostly contain natural and safe driving scenarios. As a result, current models often fail when conditioned on challenging or counterfactual trajectories-such as imperfect trajectories generated by simulators or planning systems-producing videos with severe physical inconsistencies and artifacts. To address this limitation, we propose PhyGenesis, a world model designed to generate driving videos with high visual fidelity and strong physical consistency. Our framework consists of two key components: (1) a physical condition generator that transforms potentially invalid trajectory inputs into physically plausible conditions, and (2) a physics-enhanced video generator that produces high-fidelity multi-view driving videos under these conditions. To effectively train these components, we construct a large-scale, physics-rich heterogeneous dataset. Specifically, in addition to real-world driving videos, we generate diverse challenging driving scenarios using the CARLA simulator, from which we derive supervision signals that guide the model to learn physically grounded dynamics under extreme conditions. This challenging-trajectory learning strategy enables trajectory correction and promotes physically consistent video generation. Extensive experiments demonstrate that PhyGenesis consistently outperforms state-of-the-art methods, especially on challenging trajectories. Our project page is available at: https://wm-research.github.io/PhyGenesis/.
69c49b264ae7eef815b9d9ae
Jiawei Zhou
69c49b264ae7eef815b9d9af
Zhenxin Zhu
69c49b264ae7eef815b9d9b0
Lingyi Du
69c49b264ae7eef815b9d9b1
Linye Lyu
69c49b264ae7eef815b9d9b2
Lijun Zhou
69c49b264ae7eef815b9d9b3
Zhanqian Wu
69c49b264ae7eef815b9d9b4
Hongcheng Luo
69c49b264ae7eef815b9d9b5
Zhuotao Tian
69c49b264ae7eef815b9d9b6
Bing Wang
69c49b264ae7eef815b9d9b7
Guang Chen
69c49b264ae7eef815b9d9b8
Hangjun Ye
69c49b264ae7eef815b9d9b9
Haiyang Sun
69c49b264ae7eef815b9d9ba
Yu Li
Toward Physically Consistent Driving Video World Models under Challenging Trajectories
Video generation models have shown strong potential as world models for autonomous driving simulation. However, existing approaches are primarily trained on real-world driving datasets, which mostly contain natural and safe driving scenarios. As a result, current models often fail when conditioned on challenging or counterfactual trajectories-such as imperfect trajectories generated by simulators or planning systems-producing videos with severe physical inconsistencies and artifacts. To address this limitation, we propose PhyGenesis, a world model designed to generate driving videos with high visual fidelity and strong physical consistency. Our framework consists of two key components: (1) a physical condition generator that transforms potentially invalid trajectory inputs into physically plausible conditions, and (2) a physics-enhanced video generator that produces high-fidelity multi-view driving videos under these conditions. To effectively train these components, we construct a large-scale, physics-rich heterogeneous dataset. Specifically, in addition to real-world driving videos, we generate diverse challenging driving scenarios using the CARLA simulator, from which we derive supervision signals that guide the model to learn physically grounded dynamics under extreme conditions. This challenging-trajectory learning strategy enables trajectory correction and promotes physically consistent video generation. Extensive experiments demonstrate that PhyGenesis consistently outperforms state-of-the-art methods, especially on challenging trajectories. Our project page is available at: https://wm-research.github.io/PhyGenesis/.
PhyGenesis is a world model that generates high-fidelity driving videos with physical consistency by transforming invalid trajectories into plausible conditions and using a physics-enhanced video generator trained on real and simulated driving scenarios.
world models video generation models autonomous driving simulation physical condition generator physics-enhanced video generator CARLA simulator heterogeneous dataset trajectory correction physically consistent video generation