Visual Persuasion: What Influences Decisions of Vision-Language Models?
The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.
69952bb48d17d1ee8c10ec8c
Manuel Cherep
admin_assigned
2026-02-18T13:43:04.860Z
6327a69dd907fdcccb5fab8b
/avatars/2e55406b20c472c55b6e14783a917445.svg
Manuel Cherep
mcherep
user
mcherep
69952bb48d17d1ee8c10ec8d
Pranav M R
admin_assigned
2026-02-18T13:43:11.313Z
6544c4aab4a3f3a2f490f14c
/avatars/7103d99ebed1182caa22eece4185e6d4.svg
Pranav M R
pranavmr
user
pranavmr
69952bb48d17d1ee8c10ec8e
Pattie Maes
69952bb48d17d1ee8c10ec8f
Nikhil Singh
admin_assigned
2026-02-18T13:43:17.801Z
631eef9bc1a8269da392cae8
https://cdn-avatars.huggingface.co/v1/production/uploads/1663590904331-631eef9bc1a8269da392cae8.jpeg
Nikhil Singh
nikhilsingh
user
nikhilsingh
Visual Persuasion: What Influences Decisions of Vision-Language Models?
The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.
Visual-language models' decision-making preferences are studied through controlled image choice tasks with systematic input perturbations, revealing visual vulnerabilities and safety concerns.
vision-language models visual utility revealed preference visual prompt optimization image generation model choice probability interpretability pipeline visual vulnerabilities safety concerns