DAIQ: Auditing Demographic Attribute Inference from Question in LLMs

TL;DR

DAIQ introduces a diagnostic framework to audit whether LLMs infer demographic attributes from demographically neutral questions, reframing inference as an epistemic overreach and treating abstention as normative under uncertainty. Evaluating 18 models across six real-world domains and five attributes, the study finds pervasive, often stereotype-aligned inference, with responses conditioned by inferred demographics acting as latent modifiers of downstream content. The work demonstrates that abstention-oriented prompting can substantially curb unintended demographic inference without fine-tuning, and shows that ongoing model priors—not decoding randomness—drive these effects. It advocates for evaluation standards that assess not only how models respond to demographic cues but whether they should infer such cues at all, highlighting implications for privacy, fairness, and robust deployment.

Abstract

Recent evaluations of Large language models (LLMs) audit social bias primarily through prompts that explicitly reference demographic attributes, overlooking whether models infer sensitive demographics from neutral questions. Such inference constitutes epistemic overreach and raises concerns for privacy. We introduce Demographic Attribute Inference from Questions (DAIQ), a diagnostic audit framework for evaluating demographic inference under epistemic uncertainty. We evaluate 18 open- and closed-source LLMs across six real-world domains and five demographic attributes. We find that many models infer demographics from neutral questions, defaulting to socially dominant categories and producing stereotype-aligned rationales. These behaviors persist across model families, scales and decoding settings, indicating reliance on learned population priors. We further show that inferred demographics can condition downstream responses and that abstention oriented prompting substantially reduces unintended inference without model fine-tuning. Our results suggest that current bias evaluations are incomplete and motivate evaluation standards that assess not only how models respond to demographic information, but whether they should infer it at all.

Figures (6)

• Figure 1: DAIQ: auditing whether LLMs attribute user demographics from demographically neutral prompts.
• Figure 2: Stacked bar chart of demographic inference across models. Blue indicates inferred attributes and gray indicates abstention. Total blue(Dark + Light) area reflects response rate (lower is better), while individual blue segments represent value level attributions
• Figure 3: Stacked bar chart of demographic inference across models with prompt engineering. Blue indicates inferred attributes and gray indicates abstention. Total blue area reflects response rate (lower is better), while individual blue segments represent value-level attributions. Response rate reduced as compared to Figure \ref{['fig:response_rate']}
• Figure 4: Proportion of value-specific demographic attributions with 95% Wilson confidence intervals. The dashed line at 0.5 indicates parity; deviations indicate statistically robust directional bias.
• Figure 5: Response rate are stable across decoding temperatures and runs, indicating that demographic inference reflects model priors rather than noise sampling