UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
Autonomous driving (AD) systems struggle in long-tail scenarios due to limited world knowledge and weak visual dynamic modeling. Existing vision-language-action (VLA)-based methods cannot leverage unlabeled videos for visual causal learning, while world model-based methods lack reasoning capabilities from large language models. In this paper, we construct multiple specialized datasets providing reasoning and planning annotations for complex scenarios. Then, a unified Understanding-Generation-Planning framework, named UniUGP, is proposed to synergize scene reasoning, future video generation, and trajectory planning through a hybrid expert architecture. By integrating pre-trained VLMs and video generation models, UniUGP leverages visual dynamics and semantic reasoning to enhance planning performance. Taking multi-frame observations and language instructions as input, it produces interpretable chain-of-thought reasoning, physically consistent trajectories, and coherent future videos. We introduce a four-stage training strategy that progressively builds these capabilities across multiple existing AD datasets, along with the proposed specialized datasets. Experiments demonstrate state-of-the-art performance in perception, reasoning, and decision-making, with superior generalization to challenging long-tail situations.
693a387b74fced5bf9c3241f
Hao Lu
693a387b74fced5bf9c32420
Ziyang Liu
693a387b74fced5bf9c32421
Guangfeng Jiang
admin_assigned
2025-12-11T10:49:22.527Z
660146b2bcf0790b8d73e447
/avatars/726be122297a03585c107f44f7aa0ed8.svg
jiang guangfeng
jiangxb24
user
jiangxb24
693a387b74fced5bf9c32422
Yuanfei Luo
693a387b74fced5bf9c32423
Sheng Chen
693a387b74fced5bf9c32424
Yangang Zhang
693a387b74fced5bf9c32425
Ying-Cong Chen
admin_assigned
2025-12-11T10:48:52.786Z
655cba1d87b67834000590e8
/avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
Yingcong Chen
yingcongchen
user
yingcongchen
UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
Autonomous driving (AD) systems struggle in long-tail scenarios due to limited world knowledge and weak visual dynamic modeling. Existing vision-language-action (VLA)-based methods cannot leverage unlabeled videos for visual causal learning, while world model-based methods lack reasoning capabilities from large language models. In this paper, we construct multiple specialized datasets providing reasoning and planning annotations for complex scenarios. Then, a unified Understanding-Generation-Planning framework, named UniUGP, is proposed to synergize scene reasoning, future video generation, and trajectory planning through a hybrid expert architecture. By integrating pre-trained VLMs and video generation models, UniUGP leverages visual dynamics and semantic reasoning to enhance planning performance. Taking multi-frame observations and language instructions as input, it produces interpretable chain-of-thought reasoning, physically consistent trajectories, and coherent future videos. We introduce a four-stage training strategy that progressively builds these capabilities across multiple existing AD datasets, along with the proposed specialized datasets. Experiments demonstrate state-of-the-art performance in perception, reasoning, and decision-making, with superior generalization to challenging long-tail situations.
A unified framework combines vision-language models and video generation to improve autonomous driving in complex scenarios by enhancing reasoning, trajectory planning, and video generation.
vision-language-action world model Understanding-Generation-Planning VLMs video generation models chain-of-thought reasoning physically consistent trajectories coherent future videos