How Large Language Models Systematically Misrepresent American Climate Opinions

TL;DR

The paper investigates how six large language models misrepresent U.S. climate opinions when prompted with demographic personas, by comparing AI outputs to 978 human responses across 20 CCAM questions. It reveals systematic compression of opinion diversity (variance reduced to about $0.72$ of human variance, i.e., $27.9\%$ compression) and intersectional biases, notably uniform gender assumptions that skew Black respondents differently from White and Hispanic groups. The findings show robust gender underestimation across models and complex race-by-gender-by-ideology interactions, with the largest gaps among very conservative subgroups (e.g., Black very conservatives). These biases threaten equitable climate governance by distorting whose voices are amplified or suppressed in AI-mediated policy analysis, underscoring the need for intersectional auditing and ongoing monitoring across policy contexts. The study contributes a formal, cross-model, interaction-focused framework for evaluating LLM representational fidelity against human data and discusses implications for climate research, governance, and environmental justice.

Abstract

Federal agencies and researchers increasingly use large language models to analyze and simulate public opinion. When AI mediates between the public and policymakers, accuracy across intersecting identities becomes consequential; inaccurate group-level estimates can mislead outreach, consultation, and policy design. While research examines intersectionality in LLM outputs, no study has compared these outputs against real human responses across intersecting identities. Climate policy is one such domain, and this is particularly urgent for climate change, where opinion is contested and diverse. We investigate how LLMs represent intersectional patterns in U.S. climate opinions. We prompted six LLMs with profiles of 978 respondents from a nationally representative U.S. climate opinion survey and compared AI-generated responses to actual human answers across 20 questions. We find that LLMs appear to compress the diversity of American climate opinions, predicting less-concerned groups as more concerned and vice versa. This compression is intersectional: LLMs apply uniform gender assumptions that match reality for White and Hispanic Americans but misrepresent Black Americans, where actual gender patterns differ. These patterns, which may be invisible to standard auditing approaches, could undermine equitable climate governance.

Figures (7)

• Figure 1: Overall patterns in LLM prediction mismatches reveal systematic compression of climate opinion diversity. All analyses control for question-specific and LLM model effects with clustered standard by respondent.
• Figure 2: Human responses (left) versus LLM predictions (right) by gender and race. In human data, gender patterns vary by race: White and Hispanic females report higher climate concern than males of the same race, while Black females report slightly lower concern than Black males. LLMs predict females higher than males for all three racial groups, matching reality for White and Hispanic respondents but reversing the pattern for Black respondents.
• Figure 3: Predicted LLM--human gap by political ideology, gender, and race. Positive values indicate overestimation; negative values indicate underestimation. Predictions are derived from race-stratified regression models with Gender $\times$ Ideology interactions, controlling for age, education, income, region, and religiosity (Table \ref{['tab:tab_racial_subset']} in Supplementary Information). Black and Hispanic respondents show significant but opposite gender patterns at ideological extremes: among Black very conservatives, females are overestimated relative to males (Female $\times$ Very Conservative: $\beta=+0.198$, $p<0.05$); among Hispanic very conservatives, the pattern reverses ($\beta=-0.227$, $p<0.01$). White respondents show a marginally significant negative interaction ($\beta=-0.063$, $p<0.1$). Other respondents show no significant gender differences, with wider confidence intervals reflecting smaller sample sizes.
• Figure 4: Gender underestimation persists across model architectures. GPT-4o and GPT-5 (same family) were compared to test within-family consistency; GPT-5 and Llama3 (different families) were compared to test cross-family consistency. While models diverge on political ideology, all six agree on underestimating female climate opinions.
• Figure 5: Gender $\times$ Race $\times$ Ideology interaction among very conservative respondents. Very conservatives show the largest prediction mismatches ($\beta = -0.077$), with pronounced racial heterogeneity. Black and Hispanic respondents show significant but opposite gender patterns, while White respondents show minimal gender differences.