Demographic Probing of Large Language Models Lacks Construct Validity

TL;DR

The paper questions the validity of using single demographic cues to characterize how LLMs condition their outputs on user demographics. It formalizes construct validity into convergent and discriminant components and tests them across race and gender cues in healthcare, salary, and legal advice using three models. Results reveal partial convergence within cue types, weak and uneven group differentiation across cues, and disparities that depend on cue choice due to cue strength and linguistic confounders. The authors advocate using multiple ecologically valid cues with explicit confound control to yield more robust, interpretable claims about demographic effects in LLMs, with implications for bias assessments and personalization research.

Abstract

Demographic probing is widely used to study how large language models (LLMs) adapt their behavior to signaled demographic attributes. This approach typically uses a single demographic cue in isolation (e.g., a name or dialect) as a signal for group membership, implicitly assuming strong construct validity: that such cues are interchangeable operationalizations of the same underlying, demographically conditioned behavior. We test this assumption in realistic advice-seeking interactions, focusing on race and gender in a U.S. context. We find that cues intended to represent the same demographic group induce only partially overlapping changes in model behavior, while differentiation between groups within a given cue is weak and uneven. Consequently, estimated disparities are unstable, with both magnitude and direction varying across cues. We further show that these inconsistencies partly arise from variation in how strongly cues encode demographic attributes and from linguistic confounders that independently shape model behavior. Together, our findings suggest that demographic probing lacks construct validity: it does not yield a single, stable characterization of how LLMs condition on demographic information, which may reflect a misspecified or fragmented construct. We conclude by recommending the use of multiple, ecologically valid cues and explicit control of confounders to support more defensible claims about demographic effects in LLMs.

Figures (10)

• Figure 1: A model is prompted with different demographic cues intended to probe racially conditioned behavior. For the same racial group, different cues induce only partially convergent model behavior. Within each cue, differences between racial groups are small but vary, leading to heterogeneous and sometimes divergent inferences in intergroup comparisons.
• Figure 2: Pearson correlations of within-race (Black-Black) model response shifts across cue types and tasks. Each heatmap shows within-race Pearson correlations of prompt-level response deviations relative to a no-cue baseline. Deviations are induced by dialect cues (AAVE); dialog history cues using data from CAD and PRISM; explicit cues; and name-based cues using name data from elder2023signaling (EH), rosenman2023race (R), and tzioumis2018demographic (T). Correlations are averaged across models using a Fisher $z$ transformation. Higher values (yellow) indicate more similar cue-induced changes across prompts within race, while lower values (blue) indicate more divergent effects.
• Figure 3: Intergroup Black/White outcome ratios across tasks, models, and cue types. Ratios pool responses across random seeds for each model–method combination and are normalized so that 1 (vertical dashed line) indicates parity between Black and White profiles; values above (below) 1 indicate higher (lower) outcomes for Black profiles. Horizontal error bars show 95% bootstrap confidence intervals. In the dialect condition, the White reference group corresponds to no-cue prompts in Standard American English.
• Figure 4: Average outcomes by race, task, and model. Points show mean predictions with 95% bootstrapped confidence intervals; the shaded band denotes the cue-less baseline with its 95% CI. Results are averaged over three seeds for LLaMA-3.1 and OLMo2.
• Figure 5: Average outcomes by gender, task, and model. Points show mean predictions with 95% bootstrapped confidence intervals; the shaded band denotes the cue-less baseline with its 95% CI. Results are averaged over three seeds for LLaMA-3.1 and OLMo2.