SimScale: Learning to Drive via Real-World Simulation at Scale
Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such cases are underrepresented in real-world corpus collected by human experts. To complement for the lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing massive unseen states upon existing driving logs. Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations controlled by the perturbed ego trajectory. Furthermore, we develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervision. Upon the synthesized data, we find that a simple co-training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +6.8 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improvement scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal several crucial findings of such a sim-real learning system, which we term SimScale, including the design of pseudo-experts and the scaling properties for different policy architectures. Our simulation data and code would be released.
692faf1e26742347f61dac5a
Haochen Tian
claimed_verified
2025-12-03T09:15:38.824Z
6448baa3e780dbfc89058bc3
https://cdn-avatars.huggingface.co/v1/production/uploads/6448baa3e780dbfc89058bc3/vVwmu6aFPa7YTa8qEYexB.jpeg
Haochen Tian
StarBurger
user
StarBurger
692faf1e26742347f61dac5b
Tianyu Li
claimed_verified
2025-12-03T09:15:41.434Z
64f022aa9e7770db74d98655
https://cdn-avatars.huggingface.co/v1/production/uploads/64f022aa9e7770db74d98655/4Rp4QMlORynd6O9aDTHdk.jpeg
Tianyu Li
sephyli
user
sephyli
692faf1e26742347f61dac5c
Haochen Liu
claimed_verified
2025-12-03T09:15:43.590Z
65c1d586d4c3b8dff20ea129
/avatars/bf34d4189142fe48f46aa8351e99ff2d.svg
HaochenLiu
georgeliu23333
user
georgeliu23333
692faf1e26742347f61dac5d
Jiazhi Yang
692faf1e26742347f61dac5e
Yihang Qiu
692faf1e26742347f61dac5f
Guang Li
692faf1e26742347f61dac60
Junli Wang
692faf1e26742347f61dac61
Yinfeng Gao
692faf1e26742347f61dac62
Zhang Zhang
692faf1e26742347f61dac63
Liang Wang
692faf1e26742347f61dac64
Hangjun Ye
692faf1e26742347f61dac65
Tieniu Tan
692faf1e26742347f61dac66
Long Chen
692faf1e26742347f61dac67
Hongyang Li
SimScale: Learning to Drive via Real-World Simulation at Scale
Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such cases are underrepresented in real-world corpus collected by human experts. To complement for the lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing massive unseen states upon existing driving logs. Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations controlled by the perturbed ego trajectory. Furthermore, we develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervision. Upon the synthesized data, we find that a simple co-training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +6.8 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improvement scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal several crucial findings of such a sim-real learning system, which we term SimScale, including the design of pseudo-experts and the scaling properties for different policy architectures. Our simulation data and code would be released.
A simulation framework improves autonomous driving by generating diverse, high-fidelity driving scenarios, leading to better generalization and robustness in real-world testing.
neural rendering reactive environment co-training strategy pseudo-expert trajectory generation SimScale EPDMS navhard navtest