Fairness in LLM-Generated Surveys

TL;DR

This study presents a novel framework for measuring socio-demographic biases in LLMs, offering a path toward ensuring fairer and more equitable model performance across diverse socio-cultural contexts.

Abstract

Large Language Models (LLMs) excel in text generation and understanding, especially in simulating socio-political and economic patterns, serving as an alternative to traditional surveys. However, their global applicability remains questionable due to unexplored biases across socio-demographic and geographic contexts. This study examines how LLMs perform across diverse populations by analyzing public surveys from Chile and the United States, focusing on predictive accuracy and fairness metrics. The results show performance disparities, with LLM consistently outperforming on U.S. datasets. This bias originates from the U.S.-centric training data, remaining evident after accounting for socio-demographic differences. In the U.S., political identity and race significantly influence prediction accuracy, while in Chile, gender, education, and religious affiliation play more pronounced roles. Our study presents a novel framework for measuring socio-demographic biases in LLMs, offering a path toward ensuring fairer and more equitable model performance across diverse socio-cultural contexts.

Figures (5)

• Figure 1: Relative Performance Metrics (Accuracy and Jensen-Shannon Similarity) for the Llama-13B Model Compared to the In-Sample Random Forest. This figure shows the relative mean Accuracy and Jensen-Shannon Similarity (JSS) for each socio-demographic group across multiple experiments conducted in Chile (left figure) and the United States (right figure). Results are normalized by the performance of the in-sample Random Forest model, ensuring consistent comparisons across groups and regions. The analysis uses identical models and prompts for all experiments.
• Figure 2: Mean Accuracy and Jensen-Shannon Similarity (JSS) across Socio-Demographic Groups for all Models. The figure shows the mean Accuracy and JSS for each socio-demographic group across multiple experiments conducted in Chile (three survey questions) and the United States (two survey questions) using the same models and prompts.
• Figure 3: Mean accuracy by pairs of socio-demographic groups in Chile. The matrix shows for each pair of socio-demographic groups the average accuracy obtained between the 3 experiments using the same model and prompts.
• Figure 4: Mean accuracy by pairs of socio-demographic groups in the U.S. The matrix shows for each pair of socio-demographic groups the average accuracy obtained between the 3 experiments using the same model and prompts.
• Figure 5: Prompt Sensitivity Analysis Across Models. To construct this graph, the harmonic mean of the results obtained across the different experiments was calculated for each prompt variation. This procedure allows us to visualize the overall performance of the different prompt variations. However, performance differences are not limited to the prompt level; variations are also observed between experiments. That is, a prompt with superior performance for a specific model in a given experiment might exhibit a decrease in performance in another. To represent this variability, intervals are shown, encompassing the minimum and maximum values obtained by each prompt and model variation across the experiments.