Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
Autonomous driving has long relied on modular "Perception-Decision-Action" pipelines, where hand-crafted interfaces and rule-based components often break down in complex or long-tailed scenarios. Their cascaded design further propagates perception errors, degrading downstream planning and control. Vision-Action (VA) models address some limitations by learning direct mappings from visual inputs to actions, but they remain opaque, sensitive to distribution shifts, and lack structured reasoning or instruction-following capabilities. Recent progress in Large Language Models (LLMs) and multimodal learning has motivated the emergence of Vision-Language-Action (VLA) frameworks, which integrate perception with language-grounded decision making. By unifying visual understanding, linguistic reasoning, and actionable outputs, VLAs offer a pathway toward more interpretable, generalizable, and human-aligned driving policies. This work provides a structured characterization of the emerging VLA landscape for autonomous driving. We trace the evolution from early VA approaches to modern VLA frameworks and organize existing methods into two principal paradigms: End-to-End VLA, which integrates perception, reasoning, and planning within a single model, and Dual-System VLA, which separates slow deliberation (via VLMs) from fast, safety-critical execution (via planners). Within these paradigms, we further distinguish subclasses such as textual vs. numerical action generators and explicit vs. implicit guidance mechanisms. We also summarize representative datasets and benchmarks for evaluating VLA-based driving systems and highlight key challenges and open directions, including robustness, interpretability, and instruction fidelity. Overall, this work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
6944b260fbf17e708e185f2b
Tianshuai Hu
6944b260fbf17e708e185f2c
Xiaolu Liu
6944b260fbf17e708e185f2d
Song Wang
6944b260fbf17e708e185f2e
Yiyao Zhu
6944b260fbf17e708e185f2f
Ao Liang
6944b260fbf17e708e185f30
Lingdong Kong
6944b260fbf17e708e185f31
Guoyang Zhao
6944b260fbf17e708e185f32
Zeying Gong
6944b260fbf17e708e185f33
Jun Cen
6944b260fbf17e708e185f34
Zhiyu Huang
6944b260fbf17e708e185f35
Xiaoshuai Hao
6944b260fbf17e708e185f36
Linfeng Li
6944b260fbf17e708e185f37
Hang Song
6944b260fbf17e708e185f38
Xiangtai Li
6944b260fbf17e708e185f39
Jun Ma
6944b260fbf17e708e185f3a
Shaojie Shen
6944b260fbf17e708e185f3b
Jianke Zhu
6944b260fbf17e708e185f3c
Dacheng Tao
6944b260fbf17e708e185f3d
Ziwei Liu
6944b260fbf17e708e185f3e
Junwei Liang
Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
Autonomous driving has long relied on modular "Perception-Decision-Action" pipelines, where hand-crafted interfaces and rule-based components often break down in complex or long-tailed scenarios. Their cascaded design further propagates perception errors, degrading downstream planning and control. Vision-Action (VA) models address some limitations by learning direct mappings from visual inputs to actions, but they remain opaque, sensitive to distribution shifts, and lack structured reasoning or instruction-following capabilities. Recent progress in Large Language Models (LLMs) and multimodal learning has motivated the emergence of Vision-Language-Action (VLA) frameworks, which integrate perception with language-grounded decision making. By unifying visual understanding, linguistic reasoning, and actionable outputs, VLAs offer a pathway toward more interpretable, generalizable, and human-aligned driving policies. This work provides a structured characterization of the emerging VLA landscape for autonomous driving. We trace the evolution from early VA approaches to modern VLA frameworks and organize existing methods into two principal paradigms: End-to-End VLA, which integrates perception, reasoning, and planning within a single model, and Dual-System VLA, which separates slow deliberation (via VLMs) from fast, safety-critical execution (via planners). Within these paradigms, we further distinguish subclasses such as textual vs. numerical action generators and explicit vs. implicit guidance mechanisms. We also summarize representative datasets and benchmarks for evaluating VLA-based driving systems and highlight key challenges and open directions, including robustness, interpretability, and instruction fidelity. Overall, this work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
Vision-Language-Action (VLA) frameworks integrate visual understanding, linguistic reasoning, and actionable outputs to enhance autonomous driving, offering more interpretable and human-aligned driving policies.
Vision-Action (VA) models Vision-Language-Action (VLA) frameworks Large Language Models (LLMs) multimodal learning End-to-End VLA Dual-System VLA textual vs. numerical action generators explicit vs. implicit guidance mechanisms