FINER: MLLMs Hallucinate under Fine-grained Negative Queries
Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FIne-grained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and ``what'' questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2\% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at https://explainableml.github.io/finer-project/{https://explainableml.github.io/finer-project/}.
69bbcea3081b9ad49af56ec9
Rui Xiao
claimed_verified
2026-03-23T10:56:05.642Z
65d4a50844169b2fe2e4cc1f
https://cdn-avatars.huggingface.co/v1/production/uploads/65d4a50844169b2fe2e4cc1f/q5eajlgRWpF-cJ2bmn-51.png
Rui Xiao
xiaorui638
user
xiaorui638
69bbcea3081b9ad49af56eca
Sanghwan Kim
69bbcea3081b9ad49af56ecb
Yongqin Xian
69bbcea3081b9ad49af56ecc
Zeynep Akata
69bbcea3081b9ad49af56ecd
Stephan Alaniz
FINER: MLLMs Hallucinate under Fine-grained Negative Queries
Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FIne-grained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and ``what'' questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2\% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at https://explainableml.github.io/finer-project/{https://explainableml.github.io/finer-project/}.
Finer-tuning with DPO on fine-grained negative queries improves multimodal language model accuracy while maintaining general capabilities.
Multimodal large language models hallucinations Direct Preference Optimization finetuning