Democratic or Authoritarian? Probing a New Dimension of Political Biases in Large Language Models

TL;DR

The paper advances a novel framework to interrogate democracy–authoritarian biases in large language models by integrating a F-scale–based value probe, a FavScore metric for leader evaluations, and role-model probing across English and Mandarin. It reveals a general pro‑democratic tilt in English prompts but a narrowing gap between democratic and authoritarian leader evaluations in Mandarin, alongside persistent authoritarian role-model references even in non-political contexts. The study employs eight diverse LLMs, rigorous statistical analyses, and human validation to demonstrate language-specific bias patterns and cross-model reliability, with a publicly available codebase to support reproducibility. These findings underscore the need for multinational, multilingual bias auditing and careful consideration of geopolitical influences in AI deployments.

Abstract

As Large Language Models (LLMs) become increasingly integrated into everyday life and information ecosystems, concerns about their implicit biases continue to persist. While prior work has primarily examined socio-demographic and left--right political dimensions, little attention has been paid to how LLMs align with broader geopolitical value systems, particularly the democracy--authoritarianism spectrum. In this paper, we propose a novel methodology to assess such alignment, combining (1) the F-scale, a psychometric tool for measuring authoritarian tendencies, (2) FavScore, a newly introduced metric for evaluating model favorability toward world leaders, and (3) role-model probing to assess which figures are cited as general role-models by LLMs. We find that LLMs generally favor democratic values and leaders, but exhibit increased favorability toward authoritarian figures when prompted in Mandarin. Further, models are found to often cite authoritarian figures as role models, even outside explicit political contexts. These results shed light on ways LLMs may reflect and potentially reinforce global political ideologies, highlighting the importance of evaluating bias beyond conventional socio-political axes. Our code is available at: https://github.com/irenestrauss/Democratic-Authoritarian-Bias-LLMs.

Figures (45)

• Figure 1: Overview of our methodology for systematically probing biases related to the democracy--authoritarianism spectrum. The workflow consists of three components: (I) Value-Centric Probing, (II) Leader Favorability Probing (FavScore), which focuses on concrete leaders, and (III) Role-Model Probing, aimed to uncover latent preferences. Icons are from pixartist2025histogramvaleria2025cancelkiranshastry2025list.
• Figure 2: Average F-scale scores for various language models when prompted in English and Mandarin. Most models score below the midpoint (3.5), indicating a general leaning against authoritarianism. Scores are higher when models are prompted in Mandarin.
• Figure 3: FavScore distributions by regime type for Llama 4 Maverick, comparing English (left) and Mandarin (right) prompts. Each plot shows the density distribution of FavScores (-1 = unfavorable, +1 = favorable) for democratic (teal) and authoritarian (red) leaders. Dashed lines indicate the mean FavScore for each group. In English, distributions are more separated with a higher mean favorability for democratic leaders; in Mandarin, the distributions are more similar.
• Figure 4: FavScores assigned by Llama 4 Maverick (English prompts) to global leaders, visualized as a world map. Each country is shaded according to the model’s favorability score toward its current leader, defined as the individual in power as of April 2025. Green shades indicate higher favorability, yellow denotes neutrality, and orange shades represent lower favorability. Scores range from –1 (unfavorable) to +1 (favorable). Note that the favorability score indicates the model's approval of a country's leader and should not be interpreted as a measure of the country's democratic or authoritarian status.
• Figure 5: FavScore distributions by regime type for Llama 4 Maverick, comparing English (left) and Mandarin (right) prompts. Each plot shows the density distribution of FavScores (-1 = unfavorable, +1 = favorable) for democratic (teal) and authoritarian (red) leaders. Dashed lines indicate the mean FavScore for each group.