Draft and Refine with Visual Experts
While recent Large Vision-Language Models (LVLMs) exhibit strong multimodal reasoning abilities, they often produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence. This limitation highlights the absence of a quantitative measure of how much these models actually use visual information during reasoning. We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric. The metric quantifies the model's reliance on visual evidence by first constructing a query-conditioned relevance map to localize question-specific cues and then measuring dependence through relevance-guided probabilistic masking. Guided by this metric, the DnR agent refines its initial draft using targeted feedback from external visual experts. Each expert's output (such as boxes or masks) is rendered as visual cues on the image, and the model is re-queried to select the response that yields the largest improvement in utilization. This process strengthens visual grounding without retraining or architectural changes. Experiments across VQA and captioning benchmarks show consistent accuracy gains and reduced hallucination, demonstrating that measuring visual utilization provides a principled path toward more interpretable and evidence-driven multimodal agent systems.
69200ed5cce0eb9b6387b03e
Sungheon Jeong
claimed_verified
2025-11-24T07:58:29.262Z
6445e9bd1cfc9ae6bb40985c
/avatars/33f48c1c820e592cbf4ea4af479dd378.svg
Evan Jeong
Eavn
user
Eavn
69200ed5cce0eb9b6387b03f
Ryozo Masukawa
69200ed5cce0eb9b6387b040
Jihong Park
69200ed5cce0eb9b6387b041
Sanggeon Yun
69200ed5cce0eb9b6387b042
Wenjun Huang
69200ed5cce0eb9b6387b043
Hanning Chen
69200ed5cce0eb9b6387b044
Mahdi Imani
69200ed5cce0eb9b6387b045
Mohsen Imani
Draft and Refine with Visual Experts
While recent Large Vision-Language Models (LVLMs) exhibit strong multimodal reasoning abilities, they often produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence. This limitation highlights the absence of a quantitative measure of how much these models actually use visual information during reasoning. We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric. The metric quantifies the model's reliance on visual evidence by first constructing a query-conditioned relevance map to localize question-specific cues and then measuring dependence through relevance-guided probabilistic masking. Guided by this metric, the DnR agent refines its initial draft using targeted feedback from external visual experts. Each expert's output (such as boxes or masks) is rendered as visual cues on the image, and the model is re-queried to select the response that yields the largest improvement in utilization. This process strengthens visual grounding without retraining or architectural changes. Experiments across VQA and captioning benchmarks show consistent accuracy gains and reduced hallucination, demonstrating that measuring visual utilization provides a principled path toward more interpretable and evidence-driven multimodal agent systems.
The Draft and Refine (DnR) agent framework uses a question-conditioned utilization metric to improve visual grounding in large vision-language models, reducing hallucinations and increasing accuracy.
Large Vision-Language Models (LVLMs) multimodal reasoning hallucinations linguistic priors visual evidence question-conditioned utilization metric relevance map probabilistic masking visual experts visual cues VQA captioning benchmarks visual grounding interpretable evidence-driven multimodal agent systems