Can LLMs interpret figurative language as humans do?: surface-level vs representational similarity

TL;DR

This study investigates whether instruction-tuned LLMs interpret figurative language similarly to humans, not only by comparing surface-level judgments but also by examining internal representational structure using Representational Similarity Analysis (RSA). Across 240 sentences spanning six linguistic traits, human judgments and four LLMs were collected under two prompting conditions, yielding 9,600 sentence–question evaluations. Results show robust surface-level alignment for GPT-4 but weaker representational alignment, especially for idioms and Gen Z slang, indicating that apparent human-likeness may arise from pattern-matching rather than shared semantic representations. The findings highlight the need for training and prompting regimes that cultivate context-sensitive, socially grounded semantic representations to achieve deeper human–AI interpretive alignment.

Abstract

Large language models generate judgments that resemble those of humans. Yet the extent to which these models align with human judgments in interpreting figurative and socially grounded language remains uncertain. To investigate this, human participants and four instruction-tuned LLMs of different sizes (GPT-4, Gemma-2-9B, Llama-3.2, and Mistral-7B) rated 240 dialogue-based sentences representing six linguistic traits: conventionality, sarcasm, funny, emotional, idiomacy, and slang. Each of the 240 sentences was paired with 40 interpretive questions, and both humans and LLMs rated these sentences on a 10-point Likert scale. Results indicated that humans and LLMs aligned at the surface level with humans, but diverged significantly at the representational level, especially in interpreting figurative sentences involving idioms and Gen Z slang. GPT-4 most closely approximates human representational patterns, while all models struggle with context-dependent and socio-pragmatic expressions like sarcasm, slang, and idiomacy.

Figures (5)

• Figure 1: Hypothetical semantic similarities of sentences A, B, and C.
• Figure 2: The figure shows an example of how a sentence and questions were presented to humans to rate on a 10-point Likert scale.
• Figure 3: Human participants and LLMs rated all sentences on a 10-point Likert scale. In (a), ratings were aggregated by sentence category (e.g., idiomatic sentences) for humans and models, and Pearson correlations were computed between category-level mean ratings across the 40 dimensions (questions)($1\times40$) to assess surface-level alignment. In (b), sentence-level ratings across all categories (e.g., Gen Z slang “The food is gas” and idiomatic “My arm broke me”) were used to compute pairwise Euclidean distances across 40-dimensional rating vectors, yielding a $240 \times240$ representational dissimilarity matrix. Representational alignment between humans and models was assessed by correlating the upper-triangular entries of their matrices, with correlations reported separately for each sentence category.
• Figure 4: The Surface-level similarity (a, c) and Representational similarity (b, d) analyses among humans and models for sensible and non-sensible sentences across Study 1.
• Figure 5: The Surface-level similarity (a, c) and Representational Similarity (b, d) analyses among humans and models for sensible and non-sensible sentences across Study 2.