T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation
Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language, yet its evaluation remains fragmented, often relying on unimodal metrics or narrowly scoped benchmarks that fail to capture cross-modal alignment, instruction following, and perceptual realism under complex prompts. To address this limitation, we present T2AV-Compass, a unified benchmark for comprehensive evaluation of T2AV systems, consisting of 500 diverse and complex prompts constructed via a taxonomy-driven pipeline to ensure semantic richness and physical plausibility. Besides, T2AV-Compass introduces a dual-level evaluation framework that integrates objective signal-level metrics for video quality, audio quality, and cross-modal alignment with a subjective MLLM-as-a-Judge protocol for instruction following and realism assessment. Extensive evaluation of 11 representative T2AVsystems reveals that even the strongest models fall substantially short of human-level realism and cross-modal consistency, with persistent failures in audio realism, fine-grained synchronization, instruction following, etc. These results indicate significant improvement room for future models and highlight the value of T2AV-Compass as a challenging and diagnostic testbed for advancing text-to-audio-video generation.
694ca45e746a34b55dd542de
Zhe Cao
claimed_verified
2025-12-29T14:24:36.800Z
66c7360df375ce3a32dd9fa0
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/elGBNDjw3WeV8ZEjgQ_HA.png
Zhe Cao
MichaelCaoo
user
MichaelCaoo
694ca45e746a34b55dd542df
Tao Wang
694ca45e746a34b55dd542e0
Jiaming Wang
claimed_verified
2025-12-25T20:43:43.024Z
68355c5ec0003bc40230b3f2
https://cdn-avatars.huggingface.co/v1/production/uploads/68355c5ec0003bc40230b3f2/fJjAPFtmAJskQJqxWUb-T.jpeg
jasmineWang
Jessamine
user
Jessamine
694ca45e746a34b55dd542e1
Yanghai Wang
694ca45e746a34b55dd542e2
Yuanxing Zhang
694ca45e746a34b55dd542e3
Jialu Chen
694ca45e746a34b55dd542e4
Miao Deng
694ca45e746a34b55dd542e5
Jiahao Wang
claimed_verified
2025-12-25T20:43:40.267Z
68abfd1ba1f07af43fbbf3f1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ivRfEWMAo1GQWw3x06LQp.png
jiahaowang
wang-jiahao
user
wang-jiahao
694ca45e746a34b55dd542e6
Yubin Guo
694ca45e746a34b55dd542e7
Chenxi Liao
694ca45e746a34b55dd542e8
Yize Zhang
694ca45e746a34b55dd542e9
Zhaoxiang Zhang
694ca45e746a34b55dd542ea
Jiaheng Liu
T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation
Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language, yet its evaluation remains fragmented, often relying on unimodal metrics or narrowly scoped benchmarks that fail to capture cross-modal alignment, instruction following, and perceptual realism under complex prompts. To address this limitation, we present T2AV-Compass, a unified benchmark for comprehensive evaluation of T2AV systems, consisting of 500 diverse and complex prompts constructed via a taxonomy-driven pipeline to ensure semantic richness and physical plausibility. Besides, T2AV-Compass introduces a dual-level evaluation framework that integrates objective signal-level metrics for video quality, audio quality, and cross-modal alignment with a subjective MLLM-as-a-Judge protocol for instruction following and realism assessment. Extensive evaluation of 11 representative T2AVsystems reveals that even the strongest models fall substantially short of human-level realism and cross-modal consistency, with persistent failures in audio realism, fine-grained synchronization, instruction following, etc. These results indicate significant improvement room for future models and highlight the value of T2AV-Compass as a challenging and diagnostic testbed for advancing text-to-audio-video generation.