UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Vision-Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D Vision-Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving Vision-Language-Action model based on Mixture-of-Transformers that addresses the perception-reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving. Code and model have been released at https://github.com/xiaomi-research/unidrivevla
69cf2c4de02fdd68b91b7a82
Yongkang Li
69cf2c4de02fdd68b91b7a83
Lijun Zhou
69cf2c4de02fdd68b91b7a84
Sixu Yan
69cf2c4de02fdd68b91b7a85
Bencheng Liao
69cf2c4de02fdd68b91b7a86
Tianyi Yan
69cf2c4de02fdd68b91b7a87
Kaixin Xiong
69cf2c4de02fdd68b91b7a88
Long Chen
69cf2c4de02fdd68b91b7a89
Hongwei Xie
69cf2c4de02fdd68b91b7a8a
Bing Wang
69cf2c4de02fdd68b91b7a8b
Guang Chen
69cf2c4de02fdd68b91b7a8c
Hangjun Ye
69cf2c4de02fdd68b91b7a8d
Wenyu Liu
69cf2c4de02fdd68b91b7a8e
Haiyang Sun
69cf2c4de02fdd68b91b7a8f
Xinggang Wang
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Vision-Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D Vision-Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving Vision-Language-Action model based on Mixture-of-Transformers that addresses the perception-reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving. Code and model have been released at https://github.com/xiaomi-research/unidrivevla
UniDriveVLA is a unified vision-language-action model for autonomous driving that decouples spatial perception and semantic reasoning through a mixture-of-transformers architecture with expert coordination and progressive training.
Vision-Language-Action models Mixture-of-Transformers expert decoupling masked joint attention sparse perception paradigm three-stage progressive training 3D detection motion forecasting driving-oriented VQA