Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.
6927d4bd243b2216fb75cdb6
Tianyi Xiong
claimed_verified
2025-11-28T13:48:55.333Z
6570977f87a92b76922c9950
https://cdn-avatars.huggingface.co/v1/production/uploads/6570977f87a92b76922c9950/AQGto1w6ugBvH2yCV46YU.jpeg
Tianyi Xiong
txiong23
user
txiong23
6927d4bd243b2216fb75cdb7
Yi Ge
claimed_verified
2025-11-30T20:10:24.792Z
667b04670b19955ad0f61c7e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Q0hMYkaX8WzquYCUctUKX.png
Yi Ge (Ellen)
ellenyige
user
ellenyige
6927d4bd243b2216fb75cdb8
Ming Li
6927d4bd243b2216fb75cdb9
Zuolong Zhang
6927d4bd243b2216fb75cdba
Pranav Kulkarni
claimed_verified
2025-11-30T20:10:20.233Z
6477943e950ad3b14ddb04d4
https://cdn-avatars.huggingface.co/v1/production/uploads/6477943e950ad3b14ddb04d4/0P8Uhv4RzELoIjoPC4vb4.jpeg
Pranav Kulkarni
itspranavk
user
itspranavk
6927d4bd243b2216fb75cdbb
Kaishen Wang
6927d4bd243b2216fb75cdbc
Qi He
6927d4bd243b2216fb75cdbd
Zeying Zhu
6927d4bd243b2216fb75cdbe
Chenxi Liu
6927d4bd243b2216fb75cdbf
Ruibo Chen
6927d4bd243b2216fb75cdc0
Tong Zheng
6927d4bd243b2216fb75cdc1
Yanshuo Chen
6927d4bd243b2216fb75cdc2
Xiyao Wang
claimed_verified
2025-11-28T13:48:53.739Z
655fed9fdef5905d38b84af3
/avatars/2cda4182dfd11a1e94743639e62328ea.svg
Xiyao Wang
russwang
user
russwang
6927d4bd243b2216fb75cdc3
Renrui Zhang
6927d4bd243b2216fb75cdc4
Wenhu Chen
6927d4bd243b2216fb75cdc5
Heng Huang
Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.
Multi-Crit evaluates multimodal models on following diverse criteria with metrics for pluralistic adherence, criterion-switching flexibility, and recognizing preference conflicts, revealing gaps in model capabilities.
LMMs multimodal evaluation systems instruction following human preferences multi-criterion human annotations pluralistic criteria criterion-level judgments open-ended generation verifiable reasoning tasks holistic judgment signals visual grounding reasoning fine-tuning test-time scaling boundary consistency