Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Emergence of Social Science of Large Language Models

Xiao Jia, Zhanzhan Zhao

TL;DR

This work addresses the fragmentation in the social science of large language models by constructing a reproducible, data-driven taxonomy. It combines a PRISMA-guided literature review of $270$ studies with an unsupervised machine-learning pipeline (SBERT embeddings, $K$-means clustering, UMAP, and LDA) to reveal a robust three-domain structure: LLM as Social Minds, LLM Societies, and LLM--Human Interactions. Study 2 then benchmarks this machine-derived map against an expert taxonomy, finding strong alignment (Macro-F1 ≈ $0.954$, $NMI ≈ 0.811$, $ARI ≈ 0.867$) and demonstrating that the unsupervised partitions reflect text-derived distinctions. The resulting framework provides a scalable, cross-domain lens for cumulative progress, enabling targeted theory development, methodological standardization, and living updates as the field evolves. Practically, the taxonomy guides interpretation across micro-, meso-, and human–AI interaction levels, offering a structured path toward responsible governance and integration of LLMs into social systems.

Abstract

The social science of large language models (LLMs) examines how these systems evoke mind attributions, interact with one another, and transform human activity and institutions. We conducted a systematic review of 270 studies, combining text embeddings, unsupervised clustering and topic modeling to build a computational taxonomy. Three domains emerge organically across the reviewed literature. LLM as Social Minds examines whether and when models display behaviors that elicit attributions of cognition, morality and bias, while addressing challenges such as test leakage and surface cues. LLM Societies examines multi-agent settings where interaction protocols, architectures and mechanism design shape coordination, norms, institutions and collective epistemic processes. LLM-Human Interactions examines how LLMs reshape tasks, learning, trust, work and governance, and how risks arise at the human-AI interface. This taxonomy provides a reproducible map of a fragmented field, clarifies evidentiary standards across levels of analysis, and highlights opportunities for cumulative progress in the social science of artificial intelligence.

The Emergence of Social Science of Large Language Models

TL;DR

This work addresses the fragmentation in the social science of large language models by constructing a reproducible, data-driven taxonomy. It combines a PRISMA-guided literature review of studies with an unsupervised machine-learning pipeline (SBERT embeddings, -means clustering, UMAP, and LDA) to reveal a robust three-domain structure: LLM as Social Minds, LLM Societies, and LLM--Human Interactions. Study 2 then benchmarks this machine-derived map against an expert taxonomy, finding strong alignment (Macro-F1 ≈ , , ) and demonstrating that the unsupervised partitions reflect text-derived distinctions. The resulting framework provides a scalable, cross-domain lens for cumulative progress, enabling targeted theory development, methodological standardization, and living updates as the field evolves. Practically, the taxonomy guides interpretation across micro-, meso-, and human–AI interaction levels, offering a structured path toward responsible governance and integration of LLMs into social systems.

Abstract

The social science of large language models (LLMs) examines how these systems evoke mind attributions, interact with one another, and transform human activity and institutions. We conducted a systematic review of 270 studies, combining text embeddings, unsupervised clustering and topic modeling to build a computational taxonomy. Three domains emerge organically across the reviewed literature. LLM as Social Minds examines whether and when models display behaviors that elicit attributions of cognition, morality and bias, while addressing challenges such as test leakage and surface cues. LLM Societies examines multi-agent settings where interaction protocols, architectures and mechanism design shape coordination, norms, institutions and collective epistemic processes. LLM-Human Interactions examines how LLMs reshape tasks, learning, trust, work and governance, and how risks arise at the human-AI interface. This taxonomy provides a reproducible map of a fragmented field, clarifies evidentiary standards across levels of analysis, and highlights opportunities for cumulative progress in the social science of artificial intelligence.

Paper Structure

This paper contains 29 sections, 17 figures, 4 tables.

Table of Contents

  1. Main
  2. Results
  3. Study 1: Computational Taxonomy via Unsupervised Clustering
  4. Global Structure of the Corpus
  5. Topic Modeling within Clusters
  6. Study 2: Expert-Informed Taxonomy and Human--Machine Alignment
  7. LLM as Social Minds
  8. LLM Societies
  9. LLM--Human Interactions
  10. Discussion
  11. Methods
  12. Search Strategy
  13. Screening and Eligibility
  14. Study 1
  15. Sentence embeddings and clustering
  16. ...and 14 more sections

Figures (17)

  • Figure 1: PRISMA flow diagram for the systematic review process.
  • Figure 2: Publications by Year. Annual distribution of publications in the dataset.
  • Figure 3: Top Publication Venues (Top 20). The twenty most frequent publication venues, including journals, conferences, and preprint servers.
  • Figure 4: Embedding Visualization Colored by K-Means Clusters with Geometric Medians and Confidence Ellipses. This figure shows the 2D document embeddings, the documents are colored by their assigned K-means clusters. The geometric medians of each cluster are marked by black '×' symbols, and 95% confidence ellipses are overlaid to indicate the spread of each cluster. Negative silhouette points are highlighted with red triangle markers. The x-axes and y-axes represent the Dimension 1 and Dimension 2 of the 2D UMAP projection. The axes are not semantically meaningful, but their cluster distributions reflect the structure of separation in the embedding space (see Figure \ref{['fig:llm_mindmap']} for detailed cluster meanings).
  • Figure 5: LDA Topic Word Clouds for Cluster 0 ($\lambda$ Sweep). Word clouds of LDA topics for cluster 0 across $\lambda \in \left[0,1\right]$, where $\lambda=0$ highlights topic-specific distinctive terms and $\lambda=1$ favors high-probability, globally frequent terms.
  • ...and 12 more figures