Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Metaphor Understanding Challenge Dataset for LLMs

Xiaoyu Tong, Rochelle Choenni, Martha Lewis, Ekaterina Shutova

TL;DR

The paper introduces MUNCH, a large-scale Metaphor Understanding Challenge Dataset for LLMs, to evaluate metaphor interpretation through apt paraphrase judgement and paraphrase generation. It collects over ten thousand paraphrases and about fifteen hundred inapt paraphrase triples from VUA-derived sentences across four genres, applying novelty and single-word substitution criteria to stress cross-domain reasoning over lexical cues. Two baseline models (LLaMA variants) and GPT-3.5 are evaluated, revealing substantial challenges and showing that explicit marking of metaphorical words improves performance, with results modulated by genre, novelty, and POS. The work provides a resource and analysis framework to diagnose and guide future fine-tuning and architectural strategies to enhance metaphor understanding in practical NLP tasks.

Abstract

Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/metaphor-understanding-challenge.

Metaphor Understanding Challenge Dataset for LLMs

TL;DR

The paper introduces MUNCH, a large-scale Metaphor Understanding Challenge Dataset for LLMs, to evaluate metaphor interpretation through apt paraphrase judgement and paraphrase generation. It collects over ten thousand paraphrases and about fifteen hundred inapt paraphrase triples from VUA-derived sentences across four genres, applying novelty and single-word substitution criteria to stress cross-domain reasoning over lexical cues. Two baseline models (LLaMA variants) and GPT-3.5 are evaluated, revealing substantial challenges and showing that explicit marking of metaphorical words improves performance, with results modulated by genre, novelty, and POS. The work provides a resource and analysis framework to diagnose and guide future fine-tuning and architectural strategies to enhance metaphor understanding in practical NLP tasks.

Abstract

Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/metaphor-understanding-challenge.
Paper Structure (29 sections, 8 figures, 12 tables)

This paper contains 29 sections, 8 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Data collection: metaphor samples
  4. The novelty criterion.
  5. The single-word criterion.
  6. Annotation of apt paraphrases
  7. Crowdsourcing task.
  8. Expert validation.
  9. Annotation of inapt paraphrases
  10. Inter-annotator agreement.
  11. Data analysis
  12. Model evaluation
  13. Paraphrase judgement
  14. Paraphrase generation
  15. Discussion
  16. ...and 14 more sections

Figures (8)

  • Figure 1: MUNCH dataset samples. Each metaphor sample has a $\ast$highlighted$\ast$ word that is metaphorically used, and is accompanied by up to 5 crowdsourced paraphrases: Substituting the highlighted word with one of the provided words should result in an apt paraphrase. For a selection of metaphor samples, we also provide expert annotation: a pair of correct and incorrect substitution words.
  • Figure 2: Two tasks for MUNCH: Given a sentence containing a metaphorically used word, a model is prompted to 1) select correct paraphrases from two given candidates (Paraphrase Judgement), and 2) paraphrase the sentence by replacing the highlighted metaphorically used word (Paraphrase Generation).
  • Figure 3: Distribution of the cosine similarity between target-apt, target-inapt, and apt-inapt pairs.
  • Figure 4: Example prompt for the Word-judgement task (the Implicit condition). The given sentence is shortened for illustration.
  • Figure 5: Procedure of the paraphrase generation task, using GPT-3.5 prompt and outputs as example. We first ask the model to generate a single token to get a glimpse of its top 5 answers. For each token that matches the beginning of a human answer, we let the model complete it to see whether it is a complete match.
  • ...and 3 more figures