Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Predict Associations Among Human Attitudes?

Ana Ma, Derek Powell

TL;DR

This work investigates whether frontier LLMs can reason about the latent interrelations among human attitudes beyond surface-text similarity. Using 64 Pew-derived attitude items and a sample of 376 U.S. adults, the authors assess GPT-4o’s ability to (i) reproduce the pairwise attitude correlations and (ii) predict individual responses from other attitudes across semantically similar and dissimilar prompts. GPT-4o closely mirrors human attitude correlations (about $r=0.77$, $p<0.001$) and can predict individual responses, with prediction accuracy sometimes benefiting from semantically dissimilar prompts, demonstrating genuine social inferences beyond surface similarity. The results imply that LLMs encode latent belief-system structure and can perform cross-attitudinal reasoning, while also highlighting safety concerns around personalization, manipulation, and echo chambers in AI-assisted social inference.

Abstract

Prior work has shown that large language models (LLMs) can predict human attitudes based on other attitudes, but this work has largely focused on predictions from highly similar and interrelated attitudes. In contrast, human attitudes are often strongly associated even across disparate and dissimilar topics. Using a novel dataset of human responses toward diverse attitude statements, we found that a frontier language model (GPT-4o) was able to recreate the pairwise correlations among individual attitudes and to predict individuals' attitudes from one another. Crucially, in an advance over prior work, we tested GPT-4o's ability to predict in the absence of surface-similarity between attitudes, finding that while surface similarity improves prediction accuracy, the model was still highly-capable of generating meaningful social inferences between dissimilar attitudes. Altogether, our findings indicate that LLMs capture crucial aspects of the deeper, latent structure of human belief systems.

Can Large Language Models Predict Associations Among Human Attitudes?

TL;DR

This work investigates whether frontier LLMs can reason about the latent interrelations among human attitudes beyond surface-text similarity. Using 64 Pew-derived attitude items and a sample of 376 U.S. adults, the authors assess GPT-4o’s ability to (i) reproduce the pairwise attitude correlations and (ii) predict individual responses from other attitudes across semantically similar and dissimilar prompts. GPT-4o closely mirrors human attitude correlations (about , ) and can predict individual responses, with prediction accuracy sometimes benefiting from semantically dissimilar prompts, demonstrating genuine social inferences beyond surface similarity. The results imply that LLMs encode latent belief-system structure and can perform cross-attitudinal reasoning, while also highlighting safety concerns around personalization, manipulation, and echo chambers in AI-assisted social inference.

Abstract

Prior work has shown that large language models (LLMs) can predict human attitudes based on other attitudes, but this work has largely focused on predictions from highly similar and interrelated attitudes. In contrast, human attitudes are often strongly associated even across disparate and dissimilar topics. Using a novel dataset of human responses toward diverse attitude statements, we found that a frontier language model (GPT-4o) was able to recreate the pairwise correlations among individual attitudes and to predict individuals' attitudes from one another. Crucially, in an advance over prior work, we tested GPT-4o's ability to predict in the absence of surface-similarity between attitudes, finding that while surface similarity improves prediction accuracy, the model was still highly-capable of generating meaningful social inferences between dissimilar attitudes. Altogether, our findings indicate that LLMs capture crucial aspects of the deeper, latent structure of human belief systems.

Paper Structure

This paper contains 24 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Interrelations among human attitudes and beliefs
  2. Social inference and LLM personas
  3. The present studies
  4. Study 1: Human Data Collection
  5. Participants
  6. Materials and Procedure
  7. Results
  8. Study 2: Estimating correlations with GPT-4o
  9. Model prompting
  10. Metrics
  11. GPT-4o-estimated correlations
  12. Similarity calculation
  13. Results
  14. Human and GPT-4o-estimated correlations
  15. Semantic Similarity and Estimated Correlations
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Correlations among self-reported human attitudes. Attitudes (labeled along the diagonal) are ordered by topic area (labeled on axes), and topic areas are separated by grid lines. Black bounding boxes highlight topic groupings along the diagonal indicating within-topic associations.
  • Figure 2: Scatterplot of correlation values observed in human data against correlation values among GPT-4o estimations.
  • Figure 3: Scatterplot comparing differences between rank-transformed correlation values observed in human data and estimated by GPT-4o against the cosine distance of each statement pair. A regression line highlights the negative linear relationship.