Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework

Kelaiti Xiao, Liang Yang, Dongyu Zhang, Paerhati Tulajiang, Hongfei Lin

TL;DR

Problem: idiom-based visual puns require aligning literal and figurative meanings across modalities. Approach: an iterative framework combining an LLM for prompt engineering, a T2IM for image synthesis, and an MLLM for visual understanding, with LLM-based evaluation guiding updates; built a dataset of 1,000 idioms with paired prompts and images. Findings: MLLM choice is the main driver of recognition accuracy, with GPT- or Gemini-based MLLMs performing best, while LLM choice has a smaller effect; 2-3 refinement rounds yield most gains. Significance: provides a public benchmark and insights into multimodal reasoning, suggesting emphasis on robust visual understanding over prompt generation; limitations include a single T2IM and automatic evaluation; future work expands T2IM diversity and human studies.

Abstract

We study idiom-based visual puns--images that align an idiom's literal and figurative meanings--and present an iterative framework that coordinates a large language model (LLM), a text-to-image model (T2IM), and a multimodal LLM (MLLM) for automatic generation and evaluation. Given an idiom, the system iteratively (i) generates detailed visual prompts, (ii) synthesizes an image, (iii) infers the idiom from the image, and (iv) refines the prompt until recognition succeeds or a step limit is reached. Using 1,000 idioms as inputs, we synthesize a corresponding dataset of visual pun images with paired prompts, enabling benchmarking of both generation and understanding. Experiments across 10 LLMs, 10 MLLMs, and one T2IM (Qwen-Image) show that MLLM choice is the primary performance driver: GPT achieves the highest accuracies, Gemini follows, and the best open-source MLLM (Gemma) is competitive with some closed models. On the LLM side, Claude attains the strongest average performance for prompt generation.

Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework

TL;DR

Problem: idiom-based visual puns require aligning literal and figurative meanings across modalities. Approach: an iterative framework combining an LLM for prompt engineering, a T2IM for image synthesis, and an MLLM for visual understanding, with LLM-based evaluation guiding updates; built a dataset of 1,000 idioms with paired prompts and images. Findings: MLLM choice is the main driver of recognition accuracy, with GPT- or Gemini-based MLLMs performing best, while LLM choice has a smaller effect; 2-3 refinement rounds yield most gains. Significance: provides a public benchmark and insights into multimodal reasoning, suggesting emphasis on robust visual understanding over prompt generation; limitations include a single T2IM and automatic evaluation; future work expands T2IM diversity and human studies.

Abstract

We study idiom-based visual puns--images that align an idiom's literal and figurative meanings--and present an iterative framework that coordinates a large language model (LLM), a text-to-image model (T2IM), and a multimodal LLM (MLLM) for automatic generation and evaluation. Given an idiom, the system iteratively (i) generates detailed visual prompts, (ii) synthesizes an image, (iii) infers the idiom from the image, and (iv) refines the prompt until recognition succeeds or a step limit is reached. Using 1,000 idioms as inputs, we synthesize a corresponding dataset of visual pun images with paired prompts, enabling benchmarking of both generation and understanding. Experiments across 10 LLMs, 10 MLLMs, and one T2IM (Qwen-Image) show that MLLM choice is the primary performance driver: GPT achieves the highest accuracies, Gemini follows, and the best open-source MLLM (Gemma) is competitive with some closed models. On the LLM side, Claude attains the strongest average performance for prompt generation.

Paper Structure

This paper contains 12 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Initialization
  4. Iteration
  5. Evaluation
  6. Update
  7. Experiments
  8. Experimental Setup
  9. Main Results and Analysis
  10. Ablation Study
  11. Case Study
  12. Conclusion

Figures (3)

  • Figure 1: An LLM bridges literal and metaphorical meanings of an idiom, generating prompts that guide a T2IM to create visual puns capturing both nervous feelings and literal butterflies.
  • Figure 2: Iterative pipeline for visual pun generation: the LLM crafts prompts, the T2IM synthesizes images, the MLLM infers the idiom, and the LLM evaluates and updates; solid arrows denote the current iteration, dashed arrows reference the previous iteration
  • Figure 3: Comparative visualization of the idiom "fox in a henhouse" generated by different T2IMs using identical prompting. All models were given the prompt:"Cartoon night scene at a small farm:a sly,sharp-eyed fox in asecurity guard uniform holds a keyring and clipboard labeled"Safety Check,"confidently strolling into a cozy chicken coop.Nervous hens peek from nesting boxes,one whispering to another in a speech bubble,"Isn't he supposed to protect us?""