See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.
692fa66626742347f61dab17
Le Thien Phuc Nguyen
claimed_verified
2025-12-11T10:14:08.121Z
660047c56ab19a1d21e9d764
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PFPd87Uh5Z3Y5WoCnJqIx.png
Le Thien Phuc Nguyen
plnguyen2908
user
plnguyen2908
692fa66626742347f61dab18
Zhuoran Yu
692fa66626742347f61dab19
Samuel Low Yu Hang
692fa66626742347f61dab1a
Subin An
692fa66626742347f61dab1b
Jeongik Lee
692fa66626742347f61dab1c
Yohan Ban
692fa66626742347f61dab1d
SeungEun Chung
692fa66626742347f61dab1e
Thanh-Huy Nguyen
692fa66626742347f61dab1f
JuWan Maeng
692fa66626742347f61dab20
Soochahn Lee
692fa66626742347f61dab21
Yong Jae Lee
See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.
AV-SpeakerBench is a benchmark for evaluating speaker-centric audiovisual reasoning in videos, highlighting the importance of audiovisual fusion in multimodal large language models.
multimodal large language models MLMLs audiovisual reasoning AV-SpeakerBench speaker-centric fusion-grounded expert-curated Gemini family audiovisual fusion Qwen3-Omni-30B