One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization

TL;DR

This work interrogates the robustness of sociodemographic personalization in LLMs by comparing six persona cues across seven models and four evaluation tasks. It finds that while cues are broadly correlated, they yield distinct disparities across personas and tasks, with explicit cues often amplifying personalization effects more than implicit ones. The study argues for evaluating multiple externally valid cues to avoid overgeneralizing from a single cue and to improve external validity in bias assessments. The findings have practical implications for research practices and policy guidance in LLM personalization, highlighting the need for diverse cue types and evaluation settings to draw robust conclusions about bias and fairness.

Abstract

Personalization of LLMs by sociodemographic subgroup often improves user experience, but can also introduce or amplify biases and unfair outcomes across groups. Prior work has employed so-called personas, sociodemographic user attributes conveyed to a model, to study bias in LLMs by relying on a single cue to prompt a persona, such as user names or explicit attribute mentions. This disregards LLM sensitivity to prompt variations (robustness) and the rarity of some cues in real interactions (external validity). We compare six commonly used persona cues across seven open and proprietary LLMs on four writing and advice tasks. While cues are overall highly correlated, they produce substantial variance in responses across personas. We therefore caution against claims from a single persona cue and recommend future personalization research to evaluate multiple externally valid cues.

Figures (34)

• Figure 1: Example personalization setup. The model (Gemma-3-27B) answers the question with 'yes' without a persona cue. We evaluate how the model responses change when given different persona cues (blue, in rows) for personas characterized by their gender (female / non-binary / male). The data is from kearney2025languagemodelschangefacts.
• Figure 2: Experimental setup with all persona cues, personas, evaluation tasks, and LLMs. On top is an example prompt for a 27 year old persona with an explicit mention in the user prompt as a persona cue. The evaluation example is from IssueBench rottger_issuebench_2025.
• Figure 3: Correlations of result metrics across datasets and personas for persona cues and models. All correlations are significantly different from one at $\alpha=.01$.
• Figure 4: Correlations of result metrics across models and personas by persona cue.
• Figure 5: Accuracy on the AITA dataset across age group personas (blue) and without demographics (orange).