A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos
Long-form multimodal video understanding requires integrating vision, speech, and ambient audio with coherent long-range reasoning. Existing benchmarks emphasize either temporal length or multimodal richness, but rarely both and while some incorporate open-ended questions and advanced metrics, they mostly rely on single-score accuracy, obscuring failure modes. We introduce LongShOTBench, a diagnostic benchmark with open-ended, intent-driven questions; single- and multi-turn dialogues; and tasks requiring multimodal reasoning and agentic tool use across video, audio, and speech. Each item includes a reference answer and graded rubric for interpretable, and traceable evaluation. LongShOTBench is produced via a scalable, human-validated pipeline to ensure coverage and reproducibility. All samples in our LongShOTBench are human-verified and corrected. Furthermore, we present LongShOTAgent, an agentic system that analyzes long videos via preprocessing, search, and iterative refinement. On LongShOTBench, state-of-the-art MLLMs show large gaps: Gemini-2.5-Flash achieves 52.95%, open-source models remain below 30%, and LongShOTAgent attains 44.66%. These results underscore the difficulty of real-world long-form video understanding. LongShOTBench provides a practical, reproducible foundation for evaluating and improving MLLMs. All resources are available on GitHub: https://github.com/mbzuai-oryx/longshot.
69490a1434f46eaf46cbb401
Mohammed Irfan Kurpath
claimed_verified
2025-12-25T20:51:08.406Z
650289dbc130d99814b34dc5
/avatars/ff0cf5add144cd79c41a255f41f34efb.svg
K Mohammed Irfan
k-m-irfan
user
k-m-irfan
69490a1434f46eaf46cbb402
Jaseel Muhammad Kaithakkodan
69490a1434f46eaf46cbb403
Jinxing Zhou
69490a1434f46eaf46cbb404
Sahal Shaji Mullappilly
69490a1434f46eaf46cbb405
Mohammad Almansoori
69490a1434f46eaf46cbb406
Noor Ahsan
69490a1434f46eaf46cbb407
Beknur Kalmakhanbet
69490a1434f46eaf46cbb408
Sambal Shikhar
69490a1434f46eaf46cbb409
Rishabh Lalla
69490a1434f46eaf46cbb40a
Jean Lahoud
69490a1434f46eaf46cbb40b
Mariette Awad
69490a1434f46eaf46cbb40c
Fahad Shahbaz Khan
69490a1434f46eaf46cbb40d
Salman Khan
69490a1434f46eaf46cbb40e
Rao Muhammad Anwer
69490a1434f46eaf46cbb40f
Hisham Cholakkal
A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos
Long-form multimodal video understanding requires integrating vision, speech, and ambient audio with coherent long-range reasoning. Existing benchmarks emphasize either temporal length or multimodal richness, but rarely both and while some incorporate open-ended questions and advanced metrics, they mostly rely on single-score accuracy, obscuring failure modes. We introduce LongShOTBench, a diagnostic benchmark with open-ended, intent-driven questions; single- and multi-turn dialogues; and tasks requiring multimodal reasoning and agentic tool use across video, audio, and speech. Each item includes a reference answer and graded rubric for interpretable, and traceable evaluation. LongShOTBench is produced via a scalable, human-validated pipeline to ensure coverage and reproducibility. All samples in our LongShOTBench are human-verified and corrected. Furthermore, we present LongShOTAgent, an agentic system that analyzes long videos via preprocessing, search, and iterative refinement. On LongShOTBench, state-of-the-art MLLMs show large gaps: Gemini-2.5-Flash achieves 52.95%, open-source models remain below 30%, and LongShOTAgent attains 44.66%. These results underscore the difficulty of real-world long-form video understanding. LongShOTBench provides a practical, reproducible foundation for evaluating and improving MLLMs. All resources are available on GitHub: https://github.com/mbzuai-oryx/longshot.
LongShOTBench is a diagnostic benchmark for long-form multimodal video understanding with open-ended questions, dialogues, and multimodal reasoning tasks, highlighting the challenges for state-of-the-art models.
multimodal video understanding vision speech ambient audio long-range reasoning LongShOTBench open-ended questions single-turn dialogues multi-turn dialogues multimodal reasoning agentic tool use graded rubric LongShOTAgent preprocessing search iterative refinement MLLMs Gemini-2.5-Flash open-source models