What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models
Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.
6965ae99fc8c4ecc02c7f867
Dasol Choi
claimed_verified
2026-01-13T08:24:22.667Z
66120647cac232c1507e13da
https://cdn-avatars.huggingface.co/v1/production/uploads/66120647cac232c1507e13da/iOBrsyr2mVdVLAFF6pZt6.png
DasolChoi
Dasool
user
Dasool
6965ae99fc8c4ecc02c7f868
Guijin Son
6965ae99fc8c4ecc02c7f869
Hanwool Lee
6965ae99fc8c4ecc02c7f86a
Minhyuk Kim
6965ae99fc8c4ecc02c7f86b
Hyunwoo Ko
6965ae99fc8c4ecc02c7f86c
Teabin Lim
6965ae99fc8c4ecc02c7f86d
Ahn Eungyeol
6965ae99fc8c4ecc02c7f86e
Jungwhan Kim
6965ae99fc8c4ecc02c7f86f
Seunghyeok Hong
6965ae99fc8c4ecc02c7f870
Youngsook Song
What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models
Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.
Real-world vision-language benchmarks reveal that under-specified user queries pose significant challenges for current models, with explicit query rewriting leading to substantial performance improvements.
vision-language models query explicitation real-world benchmarks visual questions user queries retrieval systems