A Computational Approach to Visual Metonymy

TL;DR

The paper tackles the understudied problem of visual metonymy by introducing a semiotics-grounded generation pipeline that leverages large language models to create representamens and visual descriptions, which are then rendered into metonymic images via text-to-image models. It presents ViMET, a 2,000-item benchmark of metonymic images paired with multiple-choice questions to probe cognitive visual reasoning in multimodal models. Empirical results show humans achieve 86.9% accuracy while vision-language models reach 65.9%, revealing a substantial gap in interpreting indirect visual cues. The work provides a publicly available dataset and a principled framework that advances the understanding and evaluation of non-literal visual reasoning, with implications for more culturally and contextually aware multimodal systems.

Abstract

Images often communicate more than they literally depict: a set of tools can suggest an occupation and a cultural artifact can suggest a tradition. This kind of indirect visual reference, known as visual metonymy, invites viewers to recover a target concept via associated cues rather than explicit depiction. In this work, we present the first computational investigation of visual metonymy. We introduce a novel pipeline grounded in semiotic theory that leverages large language models and text-to-image models to generate metonymic visual representations. Using this framework, we construct ViMET, the first visual metonymy dataset comprising 2,000 multiple-choice questions to evaluate the cognitive reasoning abilities in multimodal language models. Experimental results on our dataset reveal a significant gap between human performance (86.9%) and state-of-the-art vision-language models (65.9%), highlighting limitations in machines' ability to interpret indirect visual references. Our dataset is publicly available at: https://github.com/cincynlp/ViMET.

Figures (18)

• Figure 1: Examples of text-to-image model generated literal and metonymic images for the concept words Japan, Artist, and Acting. The literal image depicts the concept explicitly, while the metonymic image evokes the idea of the concept through cultural, contextual, or symbolic association.
• Figure 2: An illustration of the semiotic triad in the context of visual metonymy. The object (artist) is indirectly evoked through representamens (canvas, color palette, sculpture, brush). The interpretant emerges as the viewer infers the concept of an artist from the association of these visual elements without explicit depiction.
• Figure 3: Examples of different visual metonymies.
• Figure 4: Image generation pipeline. Full prompts are provided in Appendix \ref{['app_prompts_used']}.
• Figure 5: Examples of visual metonymy images generated by our pipeline.