Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models generalize analogy solving like children can?

Claire E. Stevenson, Alexandra Pafford, Han L. J. van der Maas, Melanie Mitchell

TL;DR

The study investigates whether large language models (LLMs) can generalize letter-string analogies to near (Greek) and far (Symbol) domains as humans do. By testing 42 children, 62 adults, and multiple LLMs on a controlled set of Latin, Greek, and Symbol alphabet problems with varied prompts and item variants, the authors show humans generalize well across domains, whereas LLMs exhibit robust degradation in near/far transfers. Analyses reveal that LLMs struggle with multi-rule generalization (e.g., predecessor and second successor) and rely less on flexible, on-the-fly alphabet representations, unlike children. The results suggest current LLMs lack robust, human-like analogical transfer and highlight scaling and representation gaps, with implications for evaluating AI generalization beyond surface similarity.

Abstract

In people, the ability to solve analogies such as "body : feet :: table : ?" emerges in childhood, and appears to transfer easily to other domains, such as the visual domain "( : ) :: < : ?". Recent research shows that large language models (LLMs) can solve various forms of analogies. However, can LLMs generalize analogy solving to new domains like people can? To investigate this, we had children, adults, and LLMs solve a series of letter-string analogies (e.g., a b : a c :: j k : ?) in the Latin alphabet, in a near transfer domain (Greek alphabet), and a far transfer domain (list of symbols). Children and adults easily generalized their knowledge to unfamiliar domains, whereas LLMs did not. This key difference between human and AI performance is evidence that these LLMs still struggle with robust human-like analogical transfer.

Can Large Language Models generalize analogy solving like children can?

TL;DR

The study investigates whether large language models (LLMs) can generalize letter-string analogies to near (Greek) and far (Symbol) domains as humans do. By testing 42 children, 62 adults, and multiple LLMs on a controlled set of Latin, Greek, and Symbol alphabet problems with varied prompts and item variants, the authors show humans generalize well across domains, whereas LLMs exhibit robust degradation in near/far transfers. Analyses reveal that LLMs struggle with multi-rule generalization (e.g., predecessor and second successor) and rely less on flexible, on-the-fly alphabet representations, unlike children. The results suggest current LLMs lack robust, human-like analogical transfer and highlight scaling and representation gaps, with implications for evaluating AI generalization beyond surface similarity.

Abstract

In people, the ability to solve analogies such as "body : feet :: table : ?" emerges in childhood, and appears to transfer easily to other domains, such as the visual domain "( : ) :: < : ?". Recent research shows that large language models (LLMs) can solve various forms of analogies. However, can LLMs generalize analogy solving to new domains like people can? To investigate this, we had children, adults, and LLMs solve a series of letter-string analogies (e.g., a b : a c :: j k : ?) in the Latin alphabet, in a near transfer domain (Greek alphabet), and a far transfer domain (list of symbols). Children and adults easily generalized their knowledge to unfamiliar domains, whereas LLMs did not. This key difference between human and AI performance is evidence that these LLMs still struggle with robust human-like analogical transfer.

Paper Structure

This paper contains 37 sections, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Materials
  4. Letter-String Analogy Task
  5. Rules
  6. Alphabets
  7. Human Data Collection
  8. Procedure
  9. Adults
  10. Children
  11. LLM Data Collection
  12. Procedure
  13. Prompt
  14. Pre-pending previous conversation
  15. Prompting templates
  16. ...and 22 more sections

Figures (6)

  • Figure 1: Human vs LLM performance on letter-string analogies across alphabet domains.
  • Figure 2: Letter-string analogy task item 1 in (a) baseline alphabet: Latin, (b) near transfer alphabet: Greek, and (c) far transfer alphabet: Symbol.
  • Figure 3: LLM performance by rule type across alphabet domains.
  • Figure 4: Rule-specific performance across alphabet domains for LLMs.
  • Figure 5: Mean Levenshtein string distance between incorrect and expected responses.
  • ...and 1 more figures