What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models

TL;DR

HAERAE-Vision introduces a real-world Korean multimodal benchmark of 653 under-specified visual questions and a parallel set of 653 explicit rewritten questions to quantify how missing textual context degrades vision-language model performance. Across 45 models, top performers stay below 50% accuracy on original queries, but explicit explicitation boosts results by 8–22 points, with smaller models gaining the most. Web search adds only modest gains and cannot fully compensate for underspecification, underscoring a gap between retrieval-based improvements and true intent understanding. The analysis reveals that remaining errors are largely due to Korean cultural knowledge gaps, not language or general reasoning, highlighting the need for culturally grounded models and evaluation pipelines that reflect real-world user behavior. The work also provides a scalable, checklist-based evaluation framework and robust LLM-judge reliability, offering a blueprint for assessing and improving multimodal systems in linguistically diverse, culturally rich contexts.

Abstract

Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.

Figures (9)

• Figure 1: Representative examples from HAERAE-Vision across six of the 13 domains. Each example shows an under-specified Korean question with English translation, the corresponding image, and evaluation checklist criteria. Note the informal, context-dependent nature of the original queries.
• Figure 2: Filtering pipeline showing data reduction at each stage. Numbers indicate pipeline stages described in Section \ref{['subsec:pipeline']}. The 0.76% survival rate reflects rigorous quality control. Each validated question is paired with an explicitated rewrite, yielding 1,306 query variants.
• Figure 3: Examples of query explicitation across three domains (Daily Life, Gaming, IT/Software). Original queries contain vague references that depend on images. Explicitated versions include background information to clarify the user request.
• Figure 4: Examples highlighting the cultural specificity of HAERAE-Vision: (a) Seoul subway interface, (b) traditional painting with calligraphy, (c) Korean drama scene requiring celebrity recognition, (d) TV channel settings, (e) historical family registry. Such culturally grounded items require knowledge rarely represented in English-centric datasets.
• Figure 5: Effect of query explicitation on model performance. Models are sorted by improvement magnitude. Smaller models benefit most from explicitation, with GPT-5 Nano showing +21.7 points improvement. All results averaged over 3 runs.