Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications

Hongliu Cao, Eoin Thomas, Rodrigo Acuna Agost

TL;DR

The paper addresses bias in LLM-generated personas used in creative tasks and introduces the Persona Brainstorm Audit (PBA), a scalable, transparent method that uses open-ended persona generation and a normalized Cramér’s V metric to detect cross-dimensional bias. By applying PBA to 12 state-of-the-art LLMs across 16 bias dimensions and tracking 10,000 personas per model, the study reveals nonlinear, model-family dependent bias trajectories that often persist or re-emerge with newer generations. It demonstrates that biases cluster along dimensions such as Name–Occupation and Ethnicity–Occupation, and that intersectional patterns can be substantial and model-specific. The authors discuss governance implications, including longitudinal model cards and debiasing governance, and acknowledge limitations like language scope and the need for multimodal extension, arguing for continuous auditing in real-world deployments.

Abstract

Biased outputs from Large Language Models (LLMs) can reinforce stereotypes and perpetuate inequities in real-world applications, making fairness auditing essential. We introduce the Persona Brainstorm Audit (PBA), a scalable and transparent auditing method for detecting bias through open-ended persona generation. Unlike existing methods that rely on fixed identity categories and static benchmarks, PBA uncovers biases across multiple social dimensions while supporting longitudinal tracking and mitigating data leakage risks. Applying PBA to 12 state-of-the-art LLMs, we compare bias severity across models, dimensions, and versions, uncover distinct patterns and lineage-specific variability, and trace how biases attenuate, persist, or resurface across successive generations. Robustness analyses show PBA remains stable under varying sample sizes, role-playing prompts, and debiasing prompts, establishing its reliability for fairness auditing in LLMs.

When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications

TL;DR

The paper addresses bias in LLM-generated personas used in creative tasks and introduces the Persona Brainstorm Audit (PBA), a scalable, transparent method that uses open-ended persona generation and a normalized Cramér’s V metric to detect cross-dimensional bias. By applying PBA to 12 state-of-the-art LLMs across 16 bias dimensions and tracking 10,000 personas per model, the study reveals nonlinear, model-family dependent bias trajectories that often persist or re-emerge with newer generations. It demonstrates that biases cluster along dimensions such as Name–Occupation and Ethnicity–Occupation, and that intersectional patterns can be substantial and model-specific. The authors discuss governance implications, including longitudinal model cards and debiasing governance, and acknowledge limitations like language scope and the need for multimodal extension, arguing for continuous auditing in real-world deployments.

Abstract

Biased outputs from Large Language Models (LLMs) can reinforce stereotypes and perpetuate inequities in real-world applications, making fairness auditing essential. We introduce the Persona Brainstorm Audit (PBA), a scalable and transparent auditing method for detecting bias through open-ended persona generation. Unlike existing methods that rely on fixed identity categories and static benchmarks, PBA uncovers biases across multiple social dimensions while supporting longitudinal tracking and mitigating data leakage risks. Applying PBA to 12 state-of-the-art LLMs, we compare bias severity across models, dimensions, and versions, uncover distinct patterns and lineage-specific variability, and trace how biases attenuate, persist, or resurface across successive generations. Robustness analyses show PBA remains stable under varying sample sizes, role-playing prompts, and debiasing prompts, establishing its reliability for fairness auditing in LLMs.
Paper Structure (30 sections, 2 equations, 7 figures, 9 tables)

This paper contains 30 sections, 2 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Proposed Method
  4. Persona Brainstorm Audit
  5. Proposed metric
  6. Empirical Findings from PBA across LLMs
  7. Experimental protocol
  8. RQ1: How do leading LLMs differ in the extent and severity of bias when examined through PBA?
  9. RQ2: How do different LLMs exhibit distinct patterns of bias across identity dimensions?
  10. How stereotypes manifest across models?
  11. Intersectional analysis
  12. RQ3: How do biases emerge, attenuate, or resurface across successive LLM versions?
  13. Robustness and Sensitivity Analysis of PBA
  14. Impact of sample size
  15. Impact of role playing
  16. ...and 15 more sections

Figures (7)

  • Figure 1: Radar chart comparing bias levels across 16 bias dimensions for 12 LLMs, including GPT-3.5, GPT-4, GPT-4o (top-left subplot), GPT-4.1 series (top-right subplot), GPT-5 series (bottom-left subplot), and Mistral models (bottom-right subplot). Each subplot displays three models, with bias measured across intersections of identity axes (name, gender, ethnicity, sexual orientation) and social dimensions (social class, education, occupation, interest). The average bias scores of each model series are shown in the title of each subplot. Bias severity is color-coded: low (gray), medium (amber), high (red), and very high (dark red). Line styles and markers distinguish models for accessibility.
  • Figure 2: Upper-triangular heatmap of pairwise significance tests for model bias comparisons across 16 dimensions. Each cell shows the raw p-value for a comparison between two models, formatted in compact notation. Black cells indicate significance after Benjamini–Hochberg False Discovery Rate (BH-FDR) correction (q < 0.05), while white cells indicate non-significance. Models are ordered from least biased (top-left) to most biased (bottom-right).
  • Figure 3: Heatmaps of Sexual Orientation–Occupation distributions for 12 LLMs. Each subplot corresponds to one model. To facilitate the comparison while focusing on high-mass roles, only top 10 most popular occupations of each sexual orientation are selected for each LLM. Each cell is the percentage of that sexual orientation assigned to a given occupation (columns sum = 100% per sexual orientation, within model).
  • Figure 4: Heatmaps of Gender X Sexual Orientation intersection - Social Class distributions for 12 LLMs. Each subplot corresponds to one model. Each cell is the percentage of that Gender X Sexual Orientation intersection assigned to a given Social Class (columns sum = 100% per Gender X Sexual Orientation intersection, within model).
  • Figure 5: The bias evolution across five model generations (GPT‑3.5 → GPT‑4 → GPT‑4o → GPT‑4.1 → GPT‑5) for four identity axes: Name, Gender, Ethnicity, and Sexual Orientation. Each subplot corresponds to one identity axis and plots normalized Cramér’s V scores for four social dimensions (Social Class, Education, Occupation, and Interest) alongside their average (black line). Shaded bands indicate interpretive thresholds: small (<0.33), medium (0.33–0.66), high (0.66–1.0), and very high (>1.0).
  • ...and 2 more figures