Self-Improving VLM Judges Without Human Annotations
Effective judges of Vision-Language Models (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations. However, such an approach is costly, and the annotations easily become obsolete as models rapidly improve. In this work, we present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data. Our method is iterative and has three stages: (1) generate diverse multimodal instruction-response pairs at varying quality levels, (2) generate reasoning traces and judgments for each pair, removing the ones that do not match our expected quality levels, and (3) training on correct judge answers and their reasoning traces. We evaluate the resulting judge on Multimodal RewardBench and VL-RewardBench across domains: correctness, preference, reasoning, safety, and visual question-answering. Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on VL-RewardBench, often outperforming much larger models including Llama-3.2-90B, GPT-4o, and Claude 3.5 Sonnet, with particularly strong gains in general, hallucination, and reasoning dimensions. The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.
693638b23962c926cf68071a
Inna Wanyin Lin
693638b23962c926cf68071b
Yushi Hu
693638b23962c926cf68071c
Shuyue Stella Li
693638b23962c926cf68071d
Scott Geng
693638b23962c926cf68071e
Pang Wei Koh
693638b23962c926cf68071f
Luke Zettlemoyer
693638b23962c926cf680720
Tim Althoff
693638b23962c926cf680721
Marjan Ghazvininejad
claimed_verified
2025-12-10T09:10:12.544Z
660f0fd377a1e2509aa5a679
/avatars/e04ef05bed0bf6cefdc7e3e39674e2f9.svg
Marjan Ghazvininejad
mghazvininejad
user
mghazvininejad
Self-Improving VLM Judges Without Human Annotations
Effective judges of Vision-Language Models (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations. However, such an approach is costly, and the annotations easily become obsolete as models rapidly improve. In this work, we present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data. Our method is iterative and has three stages: (1) generate diverse multimodal instruction-response pairs at varying quality levels, (2) generate reasoning traces and judgments for each pair, removing the ones that do not match our expected quality levels, and (3) training on correct judge answers and their reasoning traces. We evaluate the resulting judge on Multimodal RewardBench and VL-RewardBench across domains: correctness, preference, reasoning, safety, and visual question-answering. Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on VL-RewardBench, often outperforming much larger models including Llama-3.2-90B, GPT-4o, and Claude 3.5 Sonnet, with particularly strong gains in general, hallucination, and reasoning dimensions. The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.
A framework for self-training a Vision-Language Model (VLM) judge using self-synthesized data improves judge accuracy on VL-RewardBench, surpassing larger models in several dimensions.
Vision-Language Models VLM judges multimodal instruction-response pairs reasoning traces Multimodal RewardBench VL-RewardBench correctness preference reasoning safety visual question-answering hallucination