MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal
  LLMs
The advent of Multimodal Large Language Models (MLLMs) has expanded AI
capabilities to visual modalities, yet existing evaluation benchmarks remain
limited to single-video understanding, overlooking the critical need for
multi-video understanding in real-world scenarios (e.g., sports analytics and
autonomous driving). To address this significant gap, we introduce MVU-Eval,
the first comprehensive benchmark for evaluating Multi-Video Understanding for
MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies
through 1,824 meticulously curated question-answer pairs spanning 4,959 videos
from diverse domains, addressing both fundamental perception tasks and
high-order reasoning tasks. These capabilities are rigorously aligned with
real-world applications such as multi-sensor synthesis in autonomous systems
and cross-angle sports analytics. Through extensive evaluation of
state-of-the-art open-source and closed-source models, we reveal significant
performance discrepancies and limitations in current MLLMs' ability to perform
understanding across multiple videos. The benchmark will be made publicly
available to foster future research.
6912b9b7a644ba07c499c777
Tianhao Peng
6912b9b7a644ba07c499c778
Haochen Wang
claimed_verified
2025-11-11T19:43:43.690Z
6499809cf19fc795e7724e43
/avatars/4b3adce8c85e2f3ef05318ded6c89c3e.svg
HaochenWang
HaochenWang
user
HaochenWang
6912b9b7a644ba07c499c779
Yuanxing Zhang
6912b9b7a644ba07c499c77a
Zekun Wang
6912b9b7a644ba07c499c77b
Zili Wang
6912b9b7a644ba07c499c77c
Ge Zhang
claimed_verified
2025-11-18T14:09:39.410Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
zhangysk
user
zhangysk
6912b9b7a644ba07c499c77d
Jian Yang
6912b9b7a644ba07c499c77e
Shihao Li
claimed_verified
2025-12-04T08:51:17.393Z
67f9d060395fb1a0d7e4ae21
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GjpOfOuazN7IxcXBpVqRm.png
Shihao Li
Leexeo
user
Leexeo
6912b9b7a644ba07c499c77f
Yanghai Wang
6912b9b7a644ba07c499c780
Xintao Wang
6912b9b7a644ba07c499c781
Houyi Li
6912b9b7a644ba07c499c782
Wei Ji
6912b9b7a644ba07c499c783
Pengfei Wan
6912b9b7a644ba07c499c784
Wenhao Huang
6912b9b7a644ba07c499c785
Zhaoxiang Zhang
6912b9b7a644ba07c499c786
Jiaheng Liu
MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal
  LLMs
The advent of Multimodal Large Language Models (MLLMs) has expanded AI
capabilities to visual modalities, yet existing evaluation benchmarks remain
limited to single-video understanding, overlooking the critical need for
multi-video understanding in real-world scenarios (e.g., sports analytics and
autonomous driving). To address this significant gap, we introduce MVU-Eval,
the first comprehensive benchmark for evaluating Multi-Video Understanding for
MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies
through 1,824 meticulously curated question-answer pairs spanning 4,959 videos
from diverse domains, addressing both fundamental perception tasks and
high-order reasoning tasks. These capabilities are rigorously aligned with
real-world applications such as multi-sensor synthesis in autonomous systems
and cross-angle sports analytics. Through extensive evaluation of
state-of-the-art open-source and closed-source models, we reveal significant
performance discrepancies and limitations in current MLLMs' ability to perform
understanding across multiple videos. The benchmark will be made publicly
available to foster future research.
MVU-Eval is a comprehensive benchmark for evaluating multi-video understanding in Multimodal Large Language Models, addressing gaps in existing single-video benchmarks and highlighting performance discrepancies in real-world applications.
Multimodal Large Language Models MVU-Eval Multi-Video Understanding question-answer pairs multi-sensor synthesis cross-angle sports analytics