Vision Language Models Map Logos to Text via Semantic Entanglement in the Visual Projector

TL;DR

Logo hallucination in Vision-Language Models reveals that symbolic cues in logos trigger textual outputs even when no words are present. The authors propose a three-stage diagnostic framework—Bias Analysis, Perturbation Analysis, and Projector Diagnostics—applied across four VLMs on LogoDet-3K and Hard-60, uncovering that hallucination persists under diverse distortions and is localized to a low-dimensional projector subspace. Embedding-level ablations show removing a small set of projector directions substantially reduces hallucination with modest impact on OCR accuracy, suggesting a practical mitigation path via projector disentanglement and OCR-gated decoding. This work provides a diagnostic toolkit and actionable directions to improve the trustworthiness of multimodal systems in symbol–text intermix domains.

Abstract

Vision Language Models (VLMs) have achieved impressive progress in multimodal reasoning; yet, they remain vulnerable to hallucinations, where outputs are not grounded in visual evidence. In this paper, we investigate a previously overlooked setting: logo hallucination, where models generate brand names or textual content despite logos containing no visible words. Using curated splits of pure symbols, hybrids, and text-bearing logos, as well as the challenging Hard-60 subset, we systematically measure hallucination across leading VLMs. We further probe robustness through nine structured perturbations and show that hallucinations persist even under strong distortions, with occlusion exposing the sharpest weaknesses. Embedding-level analysis with open-weight LLaVA demonstrates that hallucination is tied to a small subset of projector dimensions, and targeted ablation substantially reduces errors while preserving OCR accuracy. Together, these findings reveal that VLMs often rely on symbolic priors rather than genuine glyph perception, particularly for iconic circular logos, and that projector subspaces play a decisive role in this failure mode. Our work contributes both a novel diagnostic lens and actionable mitigation insights, highlighting projector disentanglement and OCR-guided decoding as promising directions for building more trustworthy multimodal systems.

Figures (7)

• Figure 1: Logo-name leakage in VLMs. During training, the projector aligns logo visuals with their textual name embeddings. At inference, even text-free logos elicit brand-name outputs. Examples show leakage in both simple two-logo images and complex scenes with many logos and texts.
• Figure 2: Donut charts illustrating the relative distribution of six color categories for each model. The percentages are computed by normalizing within each model, so they represent the ratio of one color compared to the six-color total, rather than absolute contributions to the dataset. This representation makes clear that the models exhibit no systematic bias toward any specific color.
• Figure 3: Shape-wise accuracy rates across four models. Although the distributions are numerically close (23-26%), a consistent trend emerges: models hallucinate most on circular logos and least on irregular-shaped logos.
• Figure 4: Donut charts illustrating the distribution of nine perturbation categories for each evaluated VLM. Percentages are normalized within each model, so values represent the relative share of errors attributable to each perturbation type. Across models, occlusion consistently accounts for the largest proportion of failures, indicating that masking critical regions is particularly detrimental for reliable logo recognition.
• Figure 5: Embedding-level diagnosis of logo hallucination. (a) PCA projections reveal that hallucinatory outputs cluster near logo-like visual tokens, supporting the hypothesis that symbolic cues act as text triggers. (b) Ablation analysis shows that hallucination is concentrated in a small subset of projector dimensions: removing them reduces hallucination by nearly $30\%$ while only slightly lowering text-logo accuracy.