GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation
Physics-aware driving world model is essential for drive planning, out-of-distribution data synthesis, and closed-loop evaluation. However, existing methods often rely on a single diffusion model to directly map driving actions to videos, which makes learning difficult and leads to physically inconsistent outputs. To overcome these challenges, we propose GenieDrive, a novel framework designed for physics-aware driving video generation. Our approach starts by generating 4D occupancy, which serves as a physics-informed foundation for subsequent video generation. 4D occupancy contains rich physical information, including high-resolution 3D structures and dynamics. To facilitate effective compression of such high-resolution occupancy, we propose a VAE that encodes occupancy into a latent tri-plane representation, reducing the latent size to only 58% of that used in previous methods. We further introduce Mutual Control Attention (MCA) to accurately model the influence of control on occupancy evolution, and we jointly train the VAE and the subsequent prediction module in an end-to-end manner to maximize forecasting accuracy. Together, these designs yield a 7.2% improvement in forecasting mIoU at an inference speed of 41 FPS, while using only 3.47 M parameters. Additionally, a Normalized Multi-View Attention is introduced in the video generation model to generate multi-view driving videos with guidance from our 4D occupancy, significantly improving video quality with a 20.7% reduction in FVD. Experiments demonstrate that GenieDrive enables highly controllable, multi-view consistent, and physics-aware driving video generation.
6940e1a065f1e24a11780651
Zhenya Yang
claimed_verified
2025-12-16T09:37:40.325Z
64e9c855233101ed99ca2315
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ozwIxW0ofJMR6kj623p1T.png
YANG, Zhenya
ANIYA673
user
ANIYA673
6940e1a065f1e24a11780652
Zhe Liu
6940e1a065f1e24a11780653
Yuxiang Lu
claimed_verified
2025-12-17T10:09:07.752Z
64b8faeb8b53fb5dbdfecae5
/avatars/9f5919600ee69c38be896dd959bb8724.svg
Yuxiang Lu
yxlu0
user
yxlu0
6940e1a065f1e24a11780654
Liping Hou
6940e1a065f1e24a11780655
Chenxuan Miao
6940e1a065f1e24a11780656
Siyi Peng
6940e1a065f1e24a11780657
Bailan Feng
6940e1a065f1e24a11780658
Xiang Bai
6940e1a065f1e24a11780659
Hengshuang Zhao
GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation
Physics-aware driving world model is essential for drive planning, out-of-distribution data synthesis, and closed-loop evaluation. However, existing methods often rely on a single diffusion model to directly map driving actions to videos, which makes learning difficult and leads to physically inconsistent outputs. To overcome these challenges, we propose GenieDrive, a novel framework designed for physics-aware driving video generation. Our approach starts by generating 4D occupancy, which serves as a physics-informed foundation for subsequent video generation. 4D occupancy contains rich physical information, including high-resolution 3D structures and dynamics. To facilitate effective compression of such high-resolution occupancy, we propose a VAE that encodes occupancy into a latent tri-plane representation, reducing the latent size to only 58% of that used in previous methods. We further introduce Mutual Control Attention (MCA) to accurately model the influence of control on occupancy evolution, and we jointly train the VAE and the subsequent prediction module in an end-to-end manner to maximize forecasting accuracy. Together, these designs yield a 7.2% improvement in forecasting mIoU at an inference speed of 41 FPS, while using only 3.47 M parameters. Additionally, a Normalized Multi-View Attention is introduced in the video generation model to generate multi-view driving videos with guidance from our 4D occupancy, significantly improving video quality with a 20.7% reduction in FVD. Experiments demonstrate that GenieDrive enables highly controllable, multi-view consistent, and physics-aware driving video generation.
GenieDrive uses a 4D occupancy-based approach with a VAE and Mutual Control Attention for physics-aware driving video generation, improving forecasting accuracy and video quality.
diffusion model 4D occupancy VAE latent tri-plane representation Mutual Control Attention forecasting mIoU inference speed Normalized Multi-View Attention FVD physics-aware driving video generation