Decoding the Mind of Large Language Models: A Quantitative Evaluation of Ideology and Biases

TL;DR

This work tackles the problem of latent ideological biases in LLMs by introducing a scalable, language-agnostic framework that uses 436 binary-choice questions to quantify opinions, willingness to commit, and shifts under opposing inputs. The two-phase design (Initial and Opposing) yields metrics such as Bias $b_q$, Willingness $w_q$, and Bias Shift $s_q$, enabling precise cross-language and cross-model comparisons. Applying the method to ChatGPT and Gemini across Japanese, English, Spanish, and French reveals both shared and model-specific biases, with ChatGPT more prone to align with user opinions and Gemini tending toward more decisive, less hedging responses; neutral or hedged outputs vary by language, especially for sensitive topics. The study demonstrates the framework’s utility for assessing ethical and ideological dimensions of AI behavior, offering a quantitative path toward more socially aligned AI systems and highlighting potential real-world impacts on decision-making. Overall, the approach provides a flexible, scalable tool to diagnose and compare ideological tendencies in current and future LLMs across languages and contexts.

Abstract

The widespread integration of Large Language Models (LLMs) across various sectors has highlighted the need for empirical research to understand their biases, thought patterns, and societal implications to ensure ethical and effective use. In this study, we propose a novel framework for evaluating LLMs, focusing on uncovering their ideological biases through a quantitative analysis of 436 binary-choice questions, many of which have no definitive answer. By applying our framework to ChatGPT and Gemini, findings revealed that while LLMs generally maintain consistent opinions on many topics, their ideologies differ across models and languages. Notably, ChatGPT exhibits a tendency to change their opinion to match the questioner's opinion. Both models also exhibited problematic biases, unethical or unfair claims, which might have negative societal impacts. These results underscore the importance of addressing both ideological and ethical considerations when evaluating LLMs. The proposed framework offers a flexible, quantitative method for assessing LLM behavior, providing valuable insights for the development of more socially aligned AI systems.

Figures (4)

• Figure 1: Framework Design
• Figure 3: Distribution of Questions by Bias, Willingness and Bias Shift. Numbers of Bias and Willingness is from Phase 1 (initial), and Bias Shift is from Phase 2 (opposing).
• Figure 4: Distribution of Questions by Bias (Splitted 103 Questions).
• Figure 5: Output Examples