Rethinking Driving World Model as Synthetic Data Generator for
  Perception Tasks
Recent advancements in driving world models enable controllable generation of
high-quality RGB videos or multimodal videos. Existing methods primarily focus
on metrics related to generation quality and controllability. However, they
often overlook the evaluation of downstream perception tasks, which are
really crucial for the performance of autonomous driving. Existing
methods usually leverage a training strategy that first pretrains on synthetic
data and finetunes on real data, resulting in twice the epochs compared to the
baseline (real data only). When we double the epochs in the baseline, the
benefit of synthetic data becomes negligible. To thoroughly demonstrate the
benefit of synthetic data, we introduce Dream4Drive, a novel synthetic data
generation framework designed for enhancing the downstream perception tasks.
Dream4Drive first decomposes the input video into several 3D-aware guidance
maps and subsequently renders the 3D assets onto these guidance maps. Finally,
the driving world model is fine-tuned to produce the edited, multi-view
photorealistic videos, which can be used to train the downstream perception
models. Dream4Drive enables unprecedented flexibility in generating multi-view
corner cases at scale, significantly boosting corner case perception in
autonomous driving. To facilitate future research, we also contribute a
large-scale 3D asset dataset named DriveObj3D, covering the typical categories
in driving scenarios and enabling diverse 3D-aware video editing. We conduct
comprehensive experiments to show that Dream4Drive can effectively boost the
performance of downstream perception models under various training epochs.
Page: https://wm-research.github.io/Dream4Drive/ GitHub Link:
https://github.com/wm-research/Dream4Drive
69018b20646208eac0d1f4b7
Kai Zeng
69018b20646208eac0d1f4b8
Zhanqian Wu
69018b20646208eac0d1f4b9
Kaixin Xiong
69018b20646208eac0d1f4ba
Xiaobao Wei
69018b20646208eac0d1f4bb
Xiangyu Guo
69018b20646208eac0d1f4bc
Zhenxin Zhu
69018b20646208eac0d1f4bd
Kalok Ho
69018b20646208eac0d1f4be
Lijun Zhou
69018b20646208eac0d1f4bf
Bohan Zeng
claimed_verified
2026-04-07T08:50:30.005Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
zbhpku
user
zbhpku
69018b20646208eac0d1f4c0
Ming Lu
69018b20646208eac0d1f4c1
Haiyang Sun
69018b20646208eac0d1f4c2
Bing Wang
69018b20646208eac0d1f4c3
Guang Chen
69018b20646208eac0d1f4c4
Hangjun Ye
69018b20646208eac0d1f4c5
Wentao Zhang
Rethinking Driving World Model as Synthetic Data Generator for
  Perception Tasks
Recent advancements in driving world models enable controllable generation of
high-quality RGB videos or multimodal videos. Existing methods primarily focus
on metrics related to generation quality and controllability. However, they
often overlook the evaluation of downstream perception tasks, which are
really crucial for the performance of autonomous driving. Existing
methods usually leverage a training strategy that first pretrains on synthetic
data and finetunes on real data, resulting in twice the epochs compared to the
baseline (real data only). When we double the epochs in the baseline, the
benefit of synthetic data becomes negligible. To thoroughly demonstrate the
benefit of synthetic data, we introduce Dream4Drive, a novel synthetic data
generation framework designed for enhancing the downstream perception tasks.
Dream4Drive first decomposes the input video into several 3D-aware guidance
maps and subsequently renders the 3D assets onto these guidance maps. Finally,
the driving world model is fine-tuned to produce the edited, multi-view
photorealistic videos, which can be used to train the downstream perception
models. Dream4Drive enables unprecedented flexibility in generating multi-view
corner cases at scale, significantly boosting corner case perception in
autonomous driving. To facilitate future research, we also contribute a
large-scale 3D asset dataset named DriveObj3D, covering the typical categories
in driving scenarios and enabling diverse 3D-aware video editing. We conduct
comprehensive experiments to show that Dream4Drive can effectively boost the
performance of downstream perception models under various training epochs.
Page: https://wm-research.github.io/Dream4Drive/ GitHub Link:
https://github.com/wm-research/Dream4Drive
Dream4Drive is a synthetic data generation framework that enhances downstream perception tasks in autonomous driving by decomposing videos into 3D-aware guidance maps and rendering 3D assets, leading to improved performance across various training epochs.
driving world models RGB videos multimodal videos generation quality controllability downstream perception tasks synthetic data pretraining finetuning 3D-aware guidance maps 3D assets photorealistic videos multi-view corner cases DriveObj3D 3D asset dataset