Benchmarking VLMs' Reasoning About Persuasive Atypical Images

TL;DR

This work probes how vision-language models reason about atypical, persuasive images in ads by introducing three tasks (MAC, ASR, AOR) and four atypicality types (TR1, TR2, OIO, OR). It shows that off-the-shelf VLMs underperform compared to LLMs in multimodal reasoning, but an atypicality-aware verbalization strategy substantially boosts action-reason retrieval when coupled with strong LLMs. The proposed three-step pipeline—atypicality-aware verbalization, atypicality statement detection, and ARR—achieves state-of-the-art ARR results and highlights the gap in VLMs’ deep reasoning. The findings argue for incorporating atypicality inference into benchmarks and developing reasoning-rich multimodal systems to better understand persuasive imagery in advertising, with broad implications for SEO, model evaluation, and AI-assisted design.

Abstract

Vision language models (VLMs) have shown strong zero-shot generalization across various tasks, especially when integrated with large language models (LLMs). However, their ability to comprehend rhetorical and persuasive visual media, such as advertisements, remains understudied. Ads often employ atypical imagery, using surprising object juxtapositions to convey shared properties. For example, Fig. 1 (e) shows a beer with a feather-like texture. This requires advanced reasoning to deduce that this atypical representation signifies the beer's lightness. We introduce three novel tasks, Multi-label Atypicality Classification, Atypicality Statement Retrieval, and Aypical Object Recognition, to benchmark VLMs' understanding of atypicality in persuasive images. We evaluate how well VLMs use atypicality to infer an ad's message and test their reasoning abilities by employing semantically challenging negatives. Finally, we pioneer atypicality-aware verbalization by extracting comprehensive image descriptions sensitive to atypical elements. Our findings reveal that: (1) VLMs lack advanced reasoning capabilities compared to LLMs; (2) simple, effective strategies can extract atypicality-aware information, leading to comprehensive image verbalization; (3) atypicality aids persuasive advertisement understanding. Code and data will be made available.

Figures (10)

• Figure 1: Atypicality categories. We study four types of atypicality from yeinterpreting: Texture Replacement 1, Texture Replacement 2, Object Inside Objects, Object Replacement (defined in Sec. \ref{['sec:atyp-tasks']}).
• Figure 2: Atypicality Understanding and Action-Reason Retrieval Tasks. We introduce three tasks: Multi-label Atypicality Classification, Atypicality Statement Retrieval, and Atypical Object Retrieval. Incorrect/correct phrases/statements are in red/green.
• Figure 3: Atypicality definitions and atypicality relation statements.
• Figure 4: Our approach consists of three steps: (a) Image verbalization: We first prompt LLaVA to obtain top-5 objects ($V$), scene-text ($T$), scene description $IN$, and unusualness $UH$. Then we combine all the information to obtain a uniform description $\mathcal{T}_{\mathcal{V}}$. (b) Atypicality Statement Detection: We utilize $V$ and atypicality statement templates $\mathcal{S}_\mathcal{A}$ to generate the options which are then used along with $IN$ to retrieve the atypicality statement $\hat{s}$. (c) Action-Reason Retrieval: We input $\hat{s}$ along with verbalization $\mathcal{T}_\mathcal{V}$ to retrieve action-reason.
• Figure 5: ARR error analysis on Full-set.