Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial relationships within video frames and understanding the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA task. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.
693b8ec89874a2a5e4ffb312
Sunqi Fan
claimed_verified
2025-12-12T09:15:05.304Z
6434c9dc4b34368fdb07d421
https://cdn-avatars.huggingface.co/v1/production/uploads/6434c9dc4b34368fdb07d421/V_afg81iuNyMFfhM7qdgB.jpeg
fansunqi
fansunqi
user
fansunqi
693b8ec89874a2a5e4ffb313
Jiashuo Cui
693b8ec89874a2a5e4ffb314
Meng-Hao Guo
claimed_verified
2025-12-18T08:00:37.320Z
6571b51fd5c6a6d3b0ba68ad
/avatars/0ccd8fe8de857753b534356a90eb10f0.svg
gmh
menghao22
user
menghao22
693b8ec89874a2a5e4ffb315
Shuojin Yang
Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial relationships within video frames and understanding the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA task. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.
A spatiotemporal reasoning framework enhances multimodal large language models for video question answering by strategically scheduling tools to improve spatial and temporal understanding.
Multimodal Large Language Models Video Toolkit Spatiotemporal Reasoning Framework STAR VideoMME LongVideoBench GPT-4o