Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.
6980c763020e48d648c5d3ef
Linhan Wang
claimed_verified
2026-02-02T16:52:01.249Z
642ae02becec03b44648a184
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Kj0--KU5rc68MZR8_lbOI.png
Linhan Wang
LinhanWang
user
LinhanWang
6980c763020e48d648c5d3f0
Zichong Yang
6980c763020e48d648c5d3f1
Chen Bai
6980c763020e48d648c5d3f2
Guoxiang Zhang
6980c763020e48d648c5d3f3
Xiaotong Liu
6980c763020e48d648c5d3f4
Xiaoyin Zheng
6980c763020e48d648c5d3f5
Xiao-Xiao Long
6980c763020e48d648c5d3f6
Chang-Tien Lu
6980c763020e48d648c5d3f7
Cheng Lu
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.
Drive-JEPA combines V-JEPA video pretraining with multimodal trajectory distillation to achieve state-of-the-art performance in end-to-end autonomous driving.
Video Joint-Embedding Predictive Architecture V-JEPA ViT encoder trajectory distillation multimodal behaviors proposal-centric planner momentum-aware selection transformer-based decoder NAVSIM PDMS EPDMS