One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity

TL;DR

<3-5 sentence high-level summary> The paper investigates whether post-training alignment (RLHF or RLAIF) reduces the conceptual diversity of LLMs when simulating human populations in two behavioral domains. It simulates populations via temperature and persona prompting, and evaluates ten 7B open-source models against human baselines using domain-specific diversity metrics: a color-space $\Delta E$-based heterogeneity and a CRP-based clustering measure for concept representations. The main finding is that no model reaches human-level conceptual diversity, and aligned models generally show less diversity than non-aligned or instruction-tuned counterparts, with prompting exerting more influence than temperature. These results suggest a trade-off between value alignment and maintaining diversity of internal representations, with important implications for using LLMs as proxies in behavioral research.

Abstract

Researchers in social science and psychology have recently proposed using large language models (LLMs) as replacements for humans in behavioral research. In addition to arguments about whether LLMs accurately capture population-level patterns, this has raised questions about whether LLMs capture human-like conceptual diversity. Separately, it is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity. Inspired by human studies, we use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data. While no model reaches human-like diversity, aligned models generally display less diversity than their instruction fine-tuned counterparts. Our findings highlight potential trade-offs between increasing models' value alignment and decreasing the diversity of their conceptual representations.

Figures (10)

• Figure 1: We investigate LLM populations comprised of simulated individuals in two domains: color associations (top) and concept similarity (bottom). In both domains, there is both individual- and population-level variation. It is possible that individual variation overlaps with the population average (homogeneous population) or separates from it (heterogeneous population). Our experiments are designed to tease these two options apart.
• Figure 2: Human baselines in both domains. (a) Internal variability (y-axis) versus population variability (x-axis) for human participants in the word-color association domain. Reproduced from murthy_shades_2022. The measures are correlated but clustered under the identity line (the internal variability is lower than population variability), indicating a heterogeneous population in terms of conceptual diversity. (b) Probability of more than one conceptual representation, estimated using Chinese Restaurant Process model on human data for conceptual similarity judgement task marti_latent_2023.
• Figure 3: Heterogeneity of simulated LLM population in word-color association domain for prompting and temperature manipulations. The y-axis indicates $d_w$ (\ref{['eq:colors-metric']}) averaged over words. Rows $=$ baseline, followed by prompting and temperature conditions; columns $=$ model families. Darker bars indicate aligned models. For reference, the human baseline murthy_shades_2022 is 15.82.
• Figure 4: Heterogeneity of simulated LLM population in conceptual similarity domain. The y-axis indicates the probability of more than one conceptual representation, estimated using Chinese Restaurant Process model. Rows $=$ baseline, followed by prompting and temperature conditions; columns $=$ model families. Darker bars indicate aligned models. Human baselines for each conceptual category marti_latent_2023 are shown as horizontal lines.
• Figure 5: Models' and humans' color associations for a variety of word types. For words where there is high human variation (e.g. abstract words like “optimism”) the models' responses appear to collapse to just handful of colors, but these colors are easily interpretable (e.g. jealousy$\approx$green)