From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model
Temporal understanding in autonomous driving (AD) remains a significant challenge, even for recent state-of-the-art (SoTA) Vision-Language Models (VLMs). Prior work has introduced datasets and benchmarks aimed at improving temporal reasoning, but these have emphasized other video content, including sports, cooking, and movies. No existing benchmark focuses exclusively on the unique challenges of temporal understanding in ego-centric AD footage. To fill this gap, the Temporal Understanding in Autonomous Driving (TAD) benchmark is presented, which evaluates VLMs' ability to capture the dynamic relationships between actions in AD. TAD comprises nearly 6,000 question-answer (QA) pairs, spanning 7 human-designed tasks. In addition, an evaluation is performed that consists of 9 closed- and open-source generalist models as well as SoTA AD specialist models. When applied to TAD, current SoTA models demonstrated substandard accuracies, largely due to imperfect fine-grained motion understanding. To improve motion understanding and overall accuracy on TAD, two novel training-free solutions are proposed: Scene-CoT, that leverages Chain-of-Thought (CoT) and TCogMap, which incorporates an ego-centric temporal cognitive map. The proposed approaches are integrated with existing VLMs and improve average accuracy on TAD by up to 17.72%. By introducing TAD, benchmarking multiple SoTA models, and proposing effective enhancements, this work aims to catalyze future research on temporal understanding in AD. The benchmark and evaluation code are available at https://huggingface.co/datasets/vbdai/TAD{Hugging Face} and https://github.com/vbdi/tad_bench{Github}, respectively.
693717fe19d912300c34a0fb
Kevin Cannons
693717fe19d912300c34a0fc
Saeed Ranjbar Alvar
claimed_verified
2025-12-08T19:38:14.138Z
67f8267b33ef0ce3cdc24ce8
/avatars/f23e5e3ec86b450f7f29548253f217b6.svg
Saeed Ranjbar Alvar
saeedranjbar12
user
saeedranjbar12
693717fe19d912300c34a0fd
Mohammad Asiful Hossain
693717fe19d912300c34a0fe
Ahmad Rezaei
claimed_verified
2025-12-19T09:08:00.882Z
6495d9b6e6692d3676406834
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/W7ktMxTNo75rqUtTWJfCw.png
Ahmad Rezaei
AhNr
user
AhNr
693717fe19d912300c34a0ff
Mohsen Gholami
claimed_verified
2025-12-09T09:38:31.986Z
64ed16472f0bd58125027ff1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/H59YRVHXb4EUTMK5rX47k.png
Mohsen Gholami
mgholami
user
mgholami
693717fe19d912300c34a100
Alireza Heidarikhazaei
693717fe19d912300c34a101
Zhou Weimin
693717fe19d912300c34a102
Yong Zhang
693717fe19d912300c34a103
Mohammad Akbari
claimed_verified
2025-12-08T19:38:16.211Z
6545976cb8ac1a89ffa8d6cb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/fIBp6wUQ8Ln3gIDwvSv8-.png
Mohammad Akbari
moak7
user
moak7
From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model
Temporal understanding in autonomous driving (AD) remains a significant challenge, even for recent state-of-the-art (SoTA) Vision-Language Models (VLMs). Prior work has introduced datasets and benchmarks aimed at improving temporal reasoning, but these have emphasized other video content, including sports, cooking, and movies. No existing benchmark focuses exclusively on the unique challenges of temporal understanding in ego-centric AD footage. To fill this gap, the Temporal Understanding in Autonomous Driving (TAD) benchmark is presented, which evaluates VLMs' ability to capture the dynamic relationships between actions in AD. TAD comprises nearly 6,000 question-answer (QA) pairs, spanning 7 human-designed tasks. In addition, an evaluation is performed that consists of 9 closed- and open-source generalist models as well as SoTA AD specialist models. When applied to TAD, current SoTA models demonstrated substandard accuracies, largely due to imperfect fine-grained motion understanding. To improve motion understanding and overall accuracy on TAD, two novel training-free solutions are proposed: Scene-CoT, that leverages Chain-of-Thought (CoT) and TCogMap, which incorporates an ego-centric temporal cognitive map. The proposed approaches are integrated with existing VLMs and improve average accuracy on TAD by up to 17.72%. By introducing TAD, benchmarking multiple SoTA models, and proposing effective enhancements, this work aims to catalyze future research on temporal understanding in AD. The benchmark and evaluation code are available at https://huggingface.co/datasets/vbdai/TAD{Hugging Face} and https://github.com/vbdi/tad_bench{Github}, respectively.
The TAD benchmark evaluates Vision-Language Models on temporal understanding in autonomous driving footage, revealing substandard performance and proposing training-free solutions to improve accuracy.
Vision-Language Models temporal reasoning TAD benchmark question-answer pairs Chain-of-Thought TCogMap ego-centric temporal cognitive map