Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Nurse is Blue and Elephant is Rugby: Cross Domain Alignment in Large Language Models Reveal Human-like Patterns

Asaf Yehudai, Taelin Karidi, Gabriel Stanovsky, Ariel Goldstein, Omri Abend

TL;DR

This study uses a cognitive-science-inspired cross-domain alignment task to probe whether large language models exhibit human-like conceptual representations. By prompting seven instruction-following LLMs with 75 cross-domain prompts derived from a human dataset and evaluating both mappings and explanations with population- and individual-level analyses, the authors show robust above-chance alignment with human responses and explanation patterns. The findings reveal that LLMs not only produce mappings resembling typical human responses but also generate explanations that distribute across similarity categories in ways similar to humans, suggesting shared underlying representations. The work provides a framework for assessing cognitive-like behavior in LLMs and has implications for cognitive modeling, interpretability, and evaluating alignment between AI systems and human conceptual reasoning.

Abstract

Cross-domain alignment refers to the task of mapping a concept from one domain to another. For example, ``If a \textit{doctor} were a \textit{color}, what color would it be?''. This seemingly peculiar task is designed to investigate how people represent concrete and abstract concepts through their mappings between categories and their reasoning processes over those mappings. In this paper, we adapt this task from cognitive science to evaluate the conceptualization and reasoning abilities of large language models (LLMs) through a behavioral study. We examine several LLMs by prompting them with a cross-domain mapping task and analyzing their responses at both the population and individual levels. Additionally, we assess the models' ability to reason about their predictions by analyzing and categorizing their explanations for these mappings. The results reveal several similarities between humans' and models' mappings and explanations, suggesting that models represent concepts similarly to humans. This similarity is evident not only in the model representation but also in their behavior. Furthermore, the models mostly provide valid explanations and deploy reasoning paths that are similar to those of humans.

A Nurse is Blue and Elephant is Rugby: Cross Domain Alignment in Large Language Models Reveal Human-like Patterns

TL;DR

This study uses a cognitive-science-inspired cross-domain alignment task to probe whether large language models exhibit human-like conceptual representations. By prompting seven instruction-following LLMs with 75 cross-domain prompts derived from a human dataset and evaluating both mappings and explanations with population- and individual-level analyses, the authors show robust above-chance alignment with human responses and explanation patterns. The findings reveal that LLMs not only produce mappings resembling typical human responses but also generate explanations that distribute across similarity categories in ways similar to humans, suggesting shared underlying representations. The work provides a framework for assessing cognitive-like behavior in LLMs and has implications for cognitive modeling, interpretability, and evaluating alignment between AI systems and human conceptual reasoning.

Abstract

Cross-domain alignment refers to the task of mapping a concept from one domain to another. For example, ``If a \textit{doctor} were a \textit{color}, what color would it be?''. This seemingly peculiar task is designed to investigate how people represent concrete and abstract concepts through their mappings between categories and their reasoning processes over those mappings. In this paper, we adapt this task from cognitive science to evaluate the conceptualization and reasoning abilities of large language models (LLMs) through a behavioral study. We examine several LLMs by prompting them with a cross-domain mapping task and analyzing their responses at both the population and individual levels. Additionally, we assess the models' ability to reason about their predictions by analyzing and categorizing their explanations for these mappings. The results reveal several similarities between humans' and models' mappings and explanations, suggesting that models represent concepts similarly to humans. This similarity is evident not only in the model representation but also in their behavior. Furthermore, the models mostly provide valid explanations and deploy reasoning paths that are similar to those of humans.
Paper Structure (33 sections, 3 figures, 5 tables)

This paper contains 33 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Cross-Domain Mapping
  4. Experimental Setup
  5. Dataset.
  6. Models.
  7. Prompt Templates.
  8. Prompting Methodology.
  9. Metrics.
  10. Experiments $\&$ Results
  11. Population-Level Analysis.
  12. Results.
  13. Individual-Level Analysis.
  14. Results.
  15. Explanations
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Cross-Domain Alignment: Mapping object from domain A to domain B. Here, doctor from the profession domain to the color domain, and piano from the instrument domain to the animal domain.
  • Figure 2: A. An illustration of our evaluation pipeline, at the top, is the human annotation process of LL23, and at the bottom is our LLM evaluation process. B. Model-human Agreement: each bar represents the M@1 score. The dashed line represents the individual-level norm for agreement with the most popular answer. C. Llama prompt template. An example of a Llama prompt template, for the domain-pair (doctor, color).
  • Figure 3: A. A few examples of models' explanations from different categories B. Basis of cross-domain alignment, according to the model's explanations.