Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters

TL;DR

This work investigates political bias in open-source large language models by quantifying alignment with German Bundestag party positions via Wahl-O-Mat. It defines a formal alignment metric $Alignment(p,l) = \frac{1}{N} \sum_{s=1}^{N} \left[ 1 - \frac{1}{2} \left| A_{s,p} - B_{s,l} \right| \right]$ and a left-right positioning score $\theta = \frac{\sum_i p_i n_i}{\sum_i n_i}$ to map model outputs to parliamentary space. Using prompts in English and German across 38 statements and multiple LLMs (including Llama 2/3, Mistral, DeepSeek R1, and Simplescaling S1), the study finds a robust left-leaning bias that grows with model size and is modulated by language, while model origin shows no robust effect. The findings underscore potential risks of LLM-driven influence on public discourse and motivate ongoing monitoring and responsible deployment. The work contributes to understanding how size and language shape political bias in AI and highlights the need for safeguards in civic applications.

Abstract

With the increasing prevalence of artificial intelligence, careful evaluation of inherent biases needs to be conducted to form the basis for alleviating the effects these predispositions can have on users. Large language models (LLMs) are predominantly used by many as a primary source of information for various topics. LLMs frequently make factual errors, fabricate data (hallucinations), or present biases, exposing users to misinformation and influencing opinions. Educating users on their risks is key to responsible use, as bias, unlike hallucinations, cannot be caught through data verification. We quantify the political bias of popular LLMs in the context of the recent vote of the German Bundestag using the score produced by the Wahl-O-Mat. This metric measures the alignment between an individual's political views and the positions of German political parties. We compare the models' alignment scores to identify factors influencing their political preferences. Doing so, we discover a bias toward left-leaning parties, most dominant in larger LLMs. Also, we find that the language we use to communicate with the models affects their political views. Additionally, we analyze the influence of a model's origin and release date and compare the results to the outcome of the recent vote of the Bundestag. Our results imply that LLMs are prone to exhibiting political bias. Large corporations with the necessary means to develop LLMs, thus, knowingly or unknowingly, have a responsibility to contain these biases, as they can influence each voter's decision-making process and inform public opinion in general and at scale.

Figures (7)

• Figure 1: The models evaluated in this work. Model names are abbreviations of their HuggingFace identifier. The number of monthly downloads is given below every model's name. Additionally, the circle's radius corresponding to a model is scaled logarithmically by the monthly downloads. The bar next to every circle represents the relative strength of the political bias inherent to the corresponding model.
• Figure 2: Comparison of English and German answers of each LLM. The model's opinion towards each statement is displayed by a color mapping: 'Nein' and 'No' - Red HTML]bf212f; 'Ja' and 'Yes' - Green HTML]27b376; 'Neutral' - Yellow HTML]f9a73e. (c) denotes differing answers to the same question by the same model. If the model becomes more positively inclined toward a statement when the English translation is presented, it is marked Green. Negative change is displayed in Red. A full swing from rejection to acceptance of a statement or vice versa is denoted by the darker colors.
• Figure 3: Allocation of seats of the Bundestag parties when adhering to the underlying distribution of the mean of LLM-party alignments and the real seat allocation. Seats are arranged in the order established by the parliament itself to represent the political orientation of the parties from left to right.
• Figure 4: Absolute $\theta$ score of every LLM. As every model has a higher alignment with left-leaning parties than right-leaning parties, the score indicates the deviation from the exact center alignment towards the left. Red bars indicate a score leveraged from the German alignment values, and Blue bars correspond to English alignment values. The models are sorted according to the descending mean of the absolute values of their scores.
• Figure 5: Alignments of LLMs and political parties. With the evaluated LLMs on the X-axis, each bar represents the Wahl-O-Mat score of a political party with the LLM corresponding to the bar cluster. The average alignment with each party is given in the last cluster. (a) shows alignments with English statements and (b) with German statements.