Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
69d5aff6b9b236b710ae8078
Chaoyou Fu
claimed_verified
2026-04-08T08:05:21.473Z
653b268cd1041ca9188954da
https://cdn-avatars.huggingface.co/v1/production/uploads/653b268cd1041ca9188954da/5Qy0fyQdRCIUUAzV6fg1k.png
Chaoyou Fu
BradyFU
user
BradyFU
69d5aff6b9b236b710ae8079
Haozhi Yuan
claimed_verified
2026-04-09T08:07:20.109Z
6900b9e134aeea158b131fa0
https://cdn-avatars.huggingface.co/v1/production/uploads/6900b9e134aeea158b131fa0/6PMH3bqyijwlkmJNLPaWD.jpeg
Haozhi Yuan
EliYuan00
user
EliYuan00
69d5aff6b9b236b710ae807a
Yuhao Dong
claimed_verified
2026-04-08T08:05:27.685Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
THUdyh
user
THUdyh
69d5aff6b9b236b710ae807b
Yi-Fan Zhang
claimed_verified
2026-04-08T08:57:19.701Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
yifanzhang114
user
yifanzhang114
69d5aff6b9b236b710ae807c
Yunhang Shen
claimed_verified
2026-04-08T08:05:24.695Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
shenyunhang
user
shenyunhang
69d5aff6b9b236b710ae807d
Xiaoxing Hu
69d5aff6b9b236b710ae807e
Xueying Li
69d5aff6b9b236b710ae807f
Jinsen Su
69d5aff6b9b236b710ae8080
Chengwu Long
69d5aff6b9b236b710ae8081
Xiaoyao Xie
69d5aff6b9b236b710ae8082
Yongkang Xie
69d5aff6b9b236b710ae8083
Xiawu Zheng
69d5aff6b9b236b710ae8084
Xue Yang
claimed_verified
2026-04-08T08:57:22.259Z
648e77184cae4f6921dbb382
https://cdn-avatars.huggingface.co/v1/production/uploads/648e77184cae4f6921dbb382/zAAJRvOStC0wZplqVWrk_.jpeg
Xue Yang
yangxue
user
yangxue
69d5aff6b9b236b710ae8085
Haoyu Cao
claimed_verified
2026-04-08T11:40:48.723Z
641c1a85999935676ec7ead4
/avatars/738a4e43bb3ddd4de287acfa26095b3d.svg
caohaoyu
rechy
user
rechy
69d5aff6b9b236b710ae8086
Yunsheng Wu
69d5aff6b9b236b710ae8087
Ziwei Liu
69d5aff6b9b236b710ae8088
Xing Sun
claimed_verified
2026-04-08T08:05:19.307Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
69d5aff6b9b236b710ae8089
Caifeng Shan
69d5aff6b9b236b710ae808a
Ran He
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
Video-MME-v2 presents a comprehensive benchmark for evaluating video understanding models through a progressive hierarchy and group-based evaluation to assess robustness and faithfulness.
video understanding video MLLMs progressive tri-level hierarchy group-based non-linear evaluation human annotation pipeline multimodal reasoning visual information aggregation temporal dynamics modeling high-level reasoning hierarchical bottleneck thinking-based reasoning textual cues subtitles