Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Surprisal and Metaphor Novelty: Moderate Correlations and Divergent Scaling Effects

Omar Momen, Emilie Sitter, Berenike Herrmann, Sina Zarrieß

TL;DR

The paper investigates whether language-model surprisal tracks metaphor novelty across corpus-based and synthetic datasets, using 16 LM variants and a novel cloze-surprisal approach to incorporate full-sentence context. It finds significant but moderate correlations between surprisal and novelty, with divergent scaling: corpus-based datasets show inverse scaling with model size, while synthetic datasets exhibit positive scaling. Cloze-surprisal generally enhances correlation, whereas instruction-tuning yields mixed or negative effects. The results suggest surprisal is informative but limited as a standalone metric for linguistic creativity, underscoring the need for new measures and diverse datasets that capture cross-genre metaphor novelty and coherence between semantic mappings and surprisal.

Abstract

Novel metaphor comprehension involves complex semantic processes and linguistic creativity, making it an interesting task for studying language models (LMs). This study investigates whether surprisal, a probabilistic measure of predictability in LMs, correlates with different metaphor novelty datasets. We analyse surprisal from 16 LM variants on corpus-based and synthetic metaphor novelty datasets. We explore a cloze-style surprisal method that conditions on full-sentence context. Results show that LMs yield significant moderate correlations with scores/labels of metaphor novelty. We further identify divergent scaling patterns: on corpus-based data, correlation strength decreases with model size (inverse scaling effect), whereas on synthetic data it increases (Quality-Power Hypothesis). We conclude that while surprisal can partially account for annotations of metaphor novelty, it remains a limited metric of linguistic creativity.

Surprisal and Metaphor Novelty: Moderate Correlations and Divergent Scaling Effects

TL;DR

The paper investigates whether language-model surprisal tracks metaphor novelty across corpus-based and synthetic datasets, using 16 LM variants and a novel cloze-surprisal approach to incorporate full-sentence context. It finds significant but moderate correlations between surprisal and novelty, with divergent scaling: corpus-based datasets show inverse scaling with model size, while synthetic datasets exhibit positive scaling. Cloze-surprisal generally enhances correlation, whereas instruction-tuning yields mixed or negative effects. The results suggest surprisal is informative but limited as a standalone metric for linguistic creativity, underscoring the need for new measures and diverse datasets that capture cross-genre metaphor novelty and coherence between semantic mappings and surprisal.

Abstract

Novel metaphor comprehension involves complex semantic processes and linguistic creativity, making it an interesting task for studying language models (LMs). This study investigates whether surprisal, a probabilistic measure of predictability in LMs, correlates with different metaphor novelty datasets. We analyse surprisal from 16 LM variants on corpus-based and synthetic metaphor novelty datasets. We explore a cloze-style surprisal method that conditions on full-sentence context. Results show that LMs yield significant moderate correlations with scores/labels of metaphor novelty. We further identify divergent scaling patterns: on corpus-based data, correlation strength decreases with model size (inverse scaling effect), whereas on synthetic data it increases (Quality-Power Hypothesis). We conclude that while surprisal can partially account for annotations of metaphor novelty, it remains a limited metric of linguistic creativity.
Paper Structure (39 sections, 4 figures, 8 tables)

This paper contains 39 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Metaphor Annotations
  4. Predictive Powers of Surprisal
  5. LM-Based Metrics for Metaphor Novelty
  6. Metaphor Novelty Datasets
  7. Corpus‐based Datasets
  8. VUA-ratings
  9. VUA-dictionary
  10. Synthetic Datasets
  11. Lai2009
  12. GPT4o-metaphors
  13. Methods & Experiments
  14. Surprisal of a target word
  15. Cloze-surprisal
  16. ...and 24 more sections

Figures (4)

  • Figure 1: A sentence from VUA-ratings with metaphor-novelty annotations (n) and surprisal measures (s) from GPT2-base. The top part shows the direct-surprisal measure, while the bottom part shows the cloze-surprisal measure.
  • Figure 2: Effect of model size on correlation between surprisal and metaphor novelty annotations from the four datasets. Corpus-based datasets (blue) show a negative scale effect, while synthetic datasets (red) show a positive scale effect.
  • Figure 3: Histograms of metaphor novelty, and direct surprisal (dark colour) of content metaphoric words (15,155) across 10 LMs. Additionally, cloze-surprisal of the same models is plotted in light colour, and surprisal from instruct-tuned variants (whenever applicable) is plotted in lined solid colour.
  • Figure 4: Scatter plots between metaphor novelty scores on the x-axis and surprisal on the y-axis at different relative positions of metaphoric words in sentences. 0-10% means metaphoric words located at the first $1/10 * Length$ of the sentence.