Tinted Frames: Question Framing Blinds Vision-Language Models
Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.
69bd53b36f6dd7c037056ca5
Wan-Cyuan Fan
69bd53b36f6dd7c037056ca6
Jiayun Luo
69bd53b36f6dd7c037056ca7
Declan Kutscher
claimed_verified
2026-03-23T10:51:04.315Z
665bed48cd4115f092a0f750
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Muvzf256flNUAa8mBwkov.png
Declan Kutscher
d3tk
user
d3tk
69bd53b36f6dd7c037056ca8
Leonid Sigal
69bd53b36f6dd7c037056ca9
Ritwik Gupta
Tinted Frames: Question Framing Blinds Vision-Language Models
Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.
Vision-language models exhibit selective blindness where linguistic framing affects visual attention patterns, leading to reduced performance that can be improved through learnable prompt-tuning methods.
vision-language models visual attention linguistic framing attention modulation prompt-tuning learnable tokens visual grounding cross-framing consistency