LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .
692d430f4397b1ec214f696e
Zuhao Yang
claimed_verified
2025-12-01T09:10:11.311Z
6524d665ab1416594149e07e
https://cdn-avatars.huggingface.co/v1/production/uploads/6524d665ab1416594149e07e/KMsCaAtV0DLC4tqN8f2a7.png
Zuhao Yang
mwxely
user
mwxely
692d430f4397b1ec214f696f
Sudong Wang
claimed_verified
2025-12-01T09:10:14.173Z
6690f58e2f9f6f9c88e91031
https://cdn-avatars.huggingface.co/v1/production/uploads/6690f58e2f9f6f9c88e91031/QQ_VoEh7NlE6BUvii08zk.png
Sudong Wang
xiao45791
user
xiao45791
692d430f4397b1ec214f6970
Kaichen Zhang
claimed_verified
2025-12-01T10:06:41.343Z
64bb77e786e7fb5b8a317a43
https://cdn-avatars.huggingface.co/v1/production/uploads/64bb77e786e7fb5b8a317a43/J0jOrlZJ9gazdYaeSH2Bo.png
kcz
kcz358
user
kcz358
692d430f4397b1ec214f6971
Keming Wu
claimed_verified
2025-12-01T09:10:09.461Z
66bf00ca5b4e241fe266059d
https://cdn-avatars.huggingface.co/v1/production/uploads/66bf00ca5b4e241fe266059d/VoWPC_C4zoeT6dS699t7L.png
Keming Wu
wukeming11
user
wukeming11
692d430f4397b1ec214f6972
Sicong Leng
692d430f4397b1ec214f6973
Yifan Zhang
692d430f4397b1ec214f6974
Chengwei Qin
692d430f4397b1ec214f6975
Shijian Lu
692d430f4397b1ec214f6976
Xingxuan Li
claimed_verified
2026-03-19T13:05:41.942Z
6362a77dd3be91534c2e9213
https://cdn-avatars.huggingface.co/v1/production/uploads/6362a77dd3be91534c2e9213/3uUM3B1m2CFMukbkA0yDv.png
Xingxuan Li
veggiebird
user
veggiebird
692d430f4397b1ec214f6977
Lidong Bing
claimed_verified
2025-12-02T08:49:36.056Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
LidongBing
user
LidongBing
LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .
LongVT, an end-to-end framework, enhances long video reasoning by interleaving global and local analysis using multimodal tools, outperforming existing methods on challenging benchmarks.
multimodal models video reasoning textual Chain-of-Thought hallucinations long-form videos temporal grounding video cropping fine-grained question-answering VideoSIAH tool-integrated cold-start supervised fine-tuning agentic reinforcement learning agentic reinforcement fine-tuning