Hummus: A Dataset of Humorous Multimodal Metaphor Use

TL;DR

This work introduces Hummus, a richly annotated dataset of 1,000 image caption pairs from The New Yorker CapCon, focusing on humorous multimodal metaphor use. The authors propose a two-stage annotation scheme grounded in Conceptual Metaphor Theory and Incongruity Theory, enabling detailed labeling of conceptual metaphors, image-caption relations, and explanations, as well as annotations of other figurative devices. They benchmark six multimodal language models on six tasks (classification, naming, localization, caption highlighting, and explanations) and find that current models struggle to integrate visual and textual cues and to identify the underlying metaphors, often performing only slightly above random baselines. The dataset and code, including inter-annotator agreement metrics and error analyses, provide a resource for advancing multimodal humor understanding and the evaluation of multimodal reasoning systems with clear directions for improvement and future work.

Abstract

Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at github.com/xiaoyuisrain/humorous-multimodal-metaphor-use.

Figures (11)

• Figure 1: Hummus metaphor sample, with annotation for image (bounding box and label), caption (<i></i>), conceptual metaphor, explanation of how the metaphor use contributes to humor, and additional figurative devices.
• Figure 2: Metaphor sample for which multiple conceptual metaphors are annotated: pub is a coal mine; humans are animals.
• Figure 3: Conceptual metaphors identified in our dataset. Font size corresponds to frequency of occurrence.
• Figure 4: Example of (a) unidirectional and (b) bidirectional metaphorical mappings between animals and humans, and example of (c) metaphorically used idiom (go through the roof) in our dataset.
• Figure 5: Model performance in multimodal versus monomodal experiments: average F1 score for Classification, sentence similarity score for Naming and ImageLabel, Jaccard index score for CaptionHL, ROUGE-1 for Explanation. Success rate is 100% except for Qwen2-VL-7B-Instruct in the monomodal CaptionHL task (97%).