MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.
69170105a0c1cdddcec37283
Gailun Zeng
69170105a0c1cdddcec37284
Ziyang Luo
69170105a0c1cdddcec37285
Hongzhan Lin
69170105a0c1cdddcec37286
Yuchen Tian
69170105a0c1cdddcec37287
Kaixin Li
claimed_verified
2025-11-30T20:10:40.029Z
6346be8f7fb9f11870c63998
https://cdn-avatars.huggingface.co/v1/production/uploads/6346be8f7fb9f11870c63998/tFWawSkXL6bv1zgvzFWQd.png
Kaixin Li
likaixin
user
likaixin
69170105a0c1cdddcec37288
Ziyang Gong
69170105a0c1cdddcec37289
Jianxiong Guo
69170105a0c1cdddcec3728a
Jing Ma
MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.
MM-CRITIC is a benchmark for evaluating the critique abilities of Large Multimodal Models across multiple dimensions and tasks, using expert-informed ground answers for reliable scoring.
Large Multimodal Models LMMs MM-CRITIC critique ability evaluation dimensions task types expert-informed ground answers scoring rubrics GPT-4o response quality critique difficulty