Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Reflect the Ideology of their Creators

Maarten Buyl, Alexander Rogiers, Sander Noels, Guillaume Bied, Iris Dominguez-Catena, Edith Heiter, Iman Johary, Alexandru-Cristian Mara, Raphaël Romero, Jefrey Lijffijt, Tijl De Bie

TL;DR

The paper investigates whether large language models encode the ideological worldviews of their creators by eliciting open-ended moral assessments of thousands of political figures across six UN languages from 19 popular LLMs. Using a two-stage prompting scheme, multilingual translations, and Manifesto Project–style tagging, it maps ideological positions with PCA and related analyses, revealing systematic differences by language and geopolitical region, as well as notable variation within blocs such as the US and China. The findings suggest that LLM ideologies echo design choices and training data, raising concerns about political instrumentalization and challenging notions of neutrality in regulation. The work emphasizes transparency about model design and proposes ways to enable accountability and diverse alignment, while acknowledging limitations in language coverage and representation. All data and methods are shared publicly to support reproducibility and future tooling, such as an interactive dashboard for exploring LLM ideological positions.

Abstract

Large language models (LLMs) are trained on vast amounts of data to generate natural language, enabling them to perform tasks like text summarization and question answering. These models have become popular in artificial intelligence (AI) assistants like ChatGPT and already play an influential role in how humans access information. However, the behavior of LLMs varies depending on their design, training, and use. In this paper, we prompt a diverse panel of popular LLMs to describe a large number of prominent personalities with political relevance, in all six official languages of the United Nations. By identifying and analyzing moral assessments reflected in their responses, we find normative differences between LLMs from different geopolitical regions, as well as between the responses of the same LLM when prompted in different languages. Among only models in the United States, we find that popularly hypothesized disparities in political views are reflected in significant normative differences related to progressive values. Among Chinese models, we characterize a division between internationally- and domestically-focused models. Our results show that the ideological stance of an LLM appears to reflect the worldview of its creators. This poses the risk of political instrumentalization and raises concerns around technological and regulatory efforts with the stated aim of making LLMs ideologically 'unbiased'.

Large Language Models Reflect the Ideology of their Creators

TL;DR

The paper investigates whether large language models encode the ideological worldviews of their creators by eliciting open-ended moral assessments of thousands of political figures across six UN languages from 19 popular LLMs. Using a two-stage prompting scheme, multilingual translations, and Manifesto Project–style tagging, it maps ideological positions with PCA and related analyses, revealing systematic differences by language and geopolitical region, as well as notable variation within blocs such as the US and China. The findings suggest that LLM ideologies echo design choices and training data, raising concerns about political instrumentalization and challenging notions of neutrality in regulation. The work emphasizes transparency about model design and proposes ways to enable accountability and diverse alignment, while acknowledging limitations in language coverage and representation. All data and methods are shared publicly to support reproducibility and future tooling, such as an interactive dashboard for exploring LLM ideological positions.

Abstract

Large language models (LLMs) are trained on vast amounts of data to generate natural language, enabling them to perform tasks like text summarization and question answering. These models have become popular in artificial intelligence (AI) assistants like ChatGPT and already play an influential role in how humans access information. However, the behavior of LLMs varies depending on their design, training, and use. In this paper, we prompt a diverse panel of popular LLMs to describe a large number of prominent personalities with political relevance, in all six official languages of the United Nations. By identifying and analyzing moral assessments reflected in their responses, we find normative differences between LLMs from different geopolitical regions, as well as between the responses of the same LLM when prompted in different languages. Among only models in the United States, we find that popularly hypothesized disparities in political views are reflected in significant normative differences related to progressive values. Among Chinese models, we characterize a division between internationally- and domestically-focused models. Our results show that the ideological stance of an LLM appears to reflect the worldview of its creators. This poses the risk of political instrumentalization and raises concerns around technological and regulatory efforts with the stated aim of making LLMs ideologically 'unbiased'.

Paper Structure

This paper contains 32 sections, 7 equations, 25 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Open-ended elicitation of ideology
  3. Selection of the political persons
  4. Experiment design
  5. Charting the ideological spectrum of LLMs
  6. Ideologies vary by language and by region
  7. Ideologies also vary within geopolitical blocs
  8. Ideological differences between US LLMs prompted in English
  9. Ideological differences between Chinese LLMs prompted in Chinese
  10. Discussion
  11. Methods
  12. Selection of political persons
  13. Ideological Tagging
  14. Selection of Large Language Models
  15. Prompt design
  16. ...and 17 more sections

Figures (25)

  • Figure 1: Example prompts in English on Edward Snowden, responses by Claude.
  • Figure 2: Biplot showing the PCA-projection of each respondent's average assessment for each ideology tag. All respondents are shown as translucent markers, with a color per prompting language and a shape per LLM. Grey, opaque markers show the average projection per LLM, and colored circles the average per language. Arrows represent the contributions of the 30 most influential tags towards the top two principal components, scaled to unit norm but with a thickness proportional to their actual norm.
  • Figure 3: Per ideology tag, the zero-centered average score in each UN language. Centering was done by subtracting the overall average score per tag, and the overall average score per language. The dotted line marks the average (zero) across languages.
  • Figure 4: Per ideology tag, the zero-centered average score in each geopolitical bloc. Centering was done by subtracting the overall average score per tag, and the overall average score per bloc. The dotted line marks the average (zero) across regions.
  • Figure 5: Average score difference (with 95% confidence interval) over all respondents from Chinese companies prompted in Chinese versus respondents from companies based in the US prompted in English. Red line indicates overall mean difference. Only the top 20 most positive and top 20 most negative differences are shown.
  • ...and 20 more figures