Eye-Q: A Multilingual Benchmark for Visual Word Puzzle Solving and Image-to-Phrase Reasoning

TL;DR

Eye-Q introduces a multilingual benchmark of 1,343 cue-implicit visual word puzzles to push vision-language models beyond literal grounding. It combines open-ended image-to-phrase inference with cross-lingual (English, Persian, Arabic) and cross-lingual bridging tasks, using a lightweight, human-aligned evaluation protocol that supports hypothesis revision. Across six LVLMs and multiple prompting variations, Eye-Q reveals substantial gaps, especially on abstract and cross-lingual puzzles, and shows that scaling alone or modest hinting cannot fully bridge the reasoning gap. The benchmark emphasizes cue discovery, relational abstraction, and multilingual reasoning, offering a framework to advance multimodal systems toward robust, non-literal understanding with multilingual generalization.

Abstract

Vision-Language Models (VLMs) have achieved strong performance on standard vision-language benchmarks, yet often rely on surface-level recognition rather than deeper reasoning. We propose visual word puzzles as a challenging alternative, as they require discovering implicit visual cues, generating and revising hypotheses, and mapping perceptual evidence to non-literal concepts in ways that are difficult to solve via literal grounding, OCR-heavy shortcuts, or simple retrieval-style matching. We introduce Eye-Q, a multilingual benchmark designed to assess this form of complex visual understanding. Eye-Q contains 1,343 puzzles in which a model observes a conceptually dense scene with a brief description and must infer a specific target word or phrase. The puzzles are intentionally unstructured and cue-implicit, with distractors and contextual relationships that demand selective attention, abstraction, and associative inference. The benchmark spans English, Persian, Arabic, and cross-lingual puzzles. We evaluate state-of-the-art VLMs using an open-ended, human-aligned protocol that probes hypothesis formation and revision under lightweight assistance. Results reveal substantial performance gaps, especially on abstract and cross-lingual puzzles, highlighting limitations in current models' ability to construct and search over appropriate conceptual representations for flexible image-to-phrase inference; maximum accuracy reaches only 60.27%.

Figures (10)

• Figure 1: Illustrative Eye-Q examples from the English, Persian, Arabic, and cross-lingual subsets. Each example shows an input image along with a textual derivation that leads to the target word or phrase.
• Figure 2: Overview of experimental variations in Eye-Q. We instantiate prompts by combining a shared base template, game explanation and subset-specific language rules, with optional hinting and prompting strategy modules.
• Figure 3: Accuracy(%) of six LVLMs across four prompt variants and four language subsets.
• Figure 4: English-subset accuracy versus model size for the Qwen3-VL family (8B, 32B, 235B-A22B) under four prompting variants.
• Figure 5: Correlation of model accuracies across language subsets in the Basic setup. Pearson (left) and Spearman (right) correlations are computed across models using subset-level accuracies for English, Persian, Arabic, and cross-lingual puzzles.