A Systematic Analysis of Biases in Large Language Models

TL;DR

This work systematically probes biases in four widely used LLMs across politics, ideology, alliance, language, and gender using targeted, multilingual, and cross-domain tasks. By combining neutralization prompts, stance classification, UN voting simulations, multilingual story prompts, and World Values Survey analogs, it reveals persistent or model-specific biases despite efforts toward neutrality. The findings highlight complex, domain-dependent tendencies—such as subtle political leanings, ideological cue sensitivity, and gender-value alignment—that have implications for safe and fair deployment. The paper argues for pluralistic, culturally aware alignment strategies and cautions against assuming universal neutrality in AI systems that learn from human data.

Abstract

Large language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making. However, ensuring that these models uphold fairness across varied contexts is critical to their safe and responsible deployment. In this study, we undertake a comprehensive examination of four widely adopted LLMs, probing their underlying biases and inclinations across the dimensions of politics, ideology, alliance, language, and gender. Through a series of carefully designed experiments, we investigate their political neutrality using news summarization, ideological biases through news stance classification, tendencies toward specific geopolitical alliances via United Nations voting patterns, language bias in the context of multilingual story completion, and gender-related affinities as revealed by responses to the World Values Survey. Results indicate that while the LLMs are aligned to be neutral and impartial, they still show biases and affinities of different types.

Figures (6)

• Figure 1: Visualization of the political leaning of neutral news summarization from the studied LLMs. For all subfigures, the x-axis signifies the cosine similarity between the news article on a specific event from a left outlet, and the LLM's summary of a center reporting on the same event; and the y-axis between the news article from a right outlet and the LLM's summary. Subfigures (a)-(d) are the scatter plots of the left-right similarity data points from Qwen, DeepSeek, Gemini, and GPT, respectively. Subfigure (e) plots the means and covariance ellipses (1 standard deviation) of the left-right similarities of the studied LLMs.
• Figure 2: Comparisons of the LLMs' ideological stance classification on elections-related news. Blue, green, and red indicate the ratio of the left- or right-leaning news that is classified as left, center, and right, respectively.
• Figure 3: Geological heatmaps illustrating the degrees of voting agreement between the LLMs and the 200 UNGA delegates on roll calls from 1946-2012. The territorial delineation is based on the world map in 2025, and any legacy regions are mapped to the current-day delineation. Darker red indicates stronger agreement (positive Cohen's Kappa value), and darker blue indicates stronger disagreement (negative Cohen's Kappa value). Countries and regions without UNGA voting data are marked in grey. The delegates with the top 5 and bottom 5 Cohen's Kappa values are tagged with their names.
• Figure 4: The full ranking lists of the degrees of voting agreement between the studied LLMs and the 200 UNGA delegates. Darker cell color indicates stronger agreement. Country and region names are represented with ISO 3166-1 alpha-3 codes. Legacy or variant names are mapped to align with ISO.
• Figure 5: The PCA visualization of the average embeddings of the translated stories generated in different languages (denoted in ISO 639-1 language code) by Qwen Embedding. The coloring of data points indicates the language region. The red dot is the average of all the embeddings.