Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallelograms Strike Back: LLMs Generate Better Analogies than People

Qiawen Ella Liu, Raja Marjieh, Jian-Qiao Zhu, Adele E. Goldberg, Thomas L. Griffiths

Abstract

Four-term word analogies (A:B::C:D) are classically modeled geometrically as ''parallelograms,'' yet recent work suggests this model poorly captures how humans produce analogies, with simple local-similarity heuristics often providing a better account (Peterson et al., 2020). But does the parallelogram model fail because it is a bad model of analogical relations, or because people are not very good at generating relation-preserving analogies? We compared human and large language model (LLM) analogy completions on the same set of analogy problems from (Peterson et al., 2020). We find that LLM-generated analogies are reliably judged as better than human-generated ones, and are also more closely aligned with the parallelogram structure in a distributional embedding space (GloVe). Crucially, we show that the improvement over human analogies was driven by greater parallelogram alignment and reduced reliance on accessible words rather than enhanced sensitivity to local similarity. Moreover, the LLM advantage is driven not by uniformly superior responses by LLMs, but by humans producing a long tail of weak completions: when only modal (most frequent) responses by both systems are compared, the LLM advantage disappears. However, greater parallelogram alignment and lower word frequency continue to predict which LLM completions are rated higher than those of humans. Overall, these results suggest that the parallelogram model is not a poor account of word analogy. Rather, humans may often fail to produce completions that satisfy this relational constraint, whereas LLMs do so more consistently.

Parallelograms Strike Back: LLMs Generate Better Analogies than People

Abstract

Four-term word analogies (A:B::C:D) are classically modeled geometrically as ''parallelograms,'' yet recent work suggests this model poorly captures how humans produce analogies, with simple local-similarity heuristics often providing a better account (Peterson et al., 2020). But does the parallelogram model fail because it is a bad model of analogical relations, or because people are not very good at generating relation-preserving analogies? We compared human and large language model (LLM) analogy completions on the same set of analogy problems from (Peterson et al., 2020). We find that LLM-generated analogies are reliably judged as better than human-generated ones, and are also more closely aligned with the parallelogram structure in a distributional embedding space (GloVe). Crucially, we show that the improvement over human analogies was driven by greater parallelogram alignment and reduced reliance on accessible words rather than enhanced sensitivity to local similarity. Moreover, the LLM advantage is driven not by uniformly superior responses by LLMs, but by humans producing a long tail of weak completions: when only modal (most frequent) responses by both systems are compared, the LLM advantage disappears. However, greater parallelogram alignment and lower word frequency continue to predict which LLM completions are rated higher than those of humans. Overall, these results suggest that the parallelogram model is not a poor account of word analogy. Rather, humans may often fail to produce completions that satisfy this relational constraint, whereas LLMs do so more consistently.
Paper Structure (9 sections, 4 figures, 1 table)

This paper contains 9 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Do LLM Analogies Differ from Human Analogies?
  4. Are LLM Analogies Better than Human Analogies?
  5. Why are LLM Analogies better?
  6. How well do parallelogram vs. local similarities predict human vs. LLM completions?
  7. What predicts the higher ratings of LLM analogies?
  8. The LLM advantage disappears for modal responses, but what makes a good analogy does not
  9. Discussion

Figures (4)

  • Figure 1: Human–LLM similarity varies across semantic relations. Center: Similarity between human and LLM analogy completions across 20 semantic relation types from SemEval-2012. Darker colors indicate greater similarity between human and LLM responses. Insets: example analogies provided by humans and LLMs. Bar lengths show the proportion of responses; colors distinguish models and humans.
  • Figure 2: Relational similarity ratings for LLM versus human analogy completions. (a) Mean rating differences (LLM $-$ Human) by relation type for six LLMs. Dotted lines show average effects across all relations.*** $p < .001$, ** $p < .01$, * $p < .05$, n.s. = not significant.
  • Figure 3: Performance of humans and six LLMs on three GloVe-based analogy metrics. Bar charts show mean rank with 95% confidence intervals (lower is better). Insets display cumulative proportion of responses retrieved as a function of rank percentage (log scale).
  • Figure 4: (b) Standardized regression coefficients predicting rating differences. *** $p < .001$, * $p < .05$, n.s. = not significant.