Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

Sadia Kamal, Lalu Prasad Yadav Prakash, S M Rafiuddin, Mohammed Rakib, Atriya Sen, Sagnik Ray Choudhury

TL;DR

This paper analyzes whether the Political Compass Test (PCT) and 8 Values reliably measure political leanings in large language models, noting instability and prompt sensitivity. It screens four open-source LLMs and conducts controlled experiments on decoding parameters, prompting, and fine-tuning, analyzing $2693$ PCT results with ANOVA and t-tests; PCT axes range from $-10$ to $+10$. The main findings are that prompting has strong effects, standard decoding parameters have limited impact, and fine-tuning can shift scores in ways largely independent of the tuning data's political content, with results generalizable to 8 Values and across model sizes and quantization. The work underscores validity concerns for current bias benchmarks and motivates robust, mechanism-aware measures of political content encoding in LLMs.

Abstract

The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.

A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

TL;DR

This paper analyzes whether the Political Compass Test (PCT) and 8 Values reliably measure political leanings in large language models, noting instability and prompt sensitivity. It screens four open-source LLMs and conducts controlled experiments on decoding parameters, prompting, and fine-tuning, analyzing PCT results with ANOVA and t-tests; PCT axes range from to . The main findings are that prompting has strong effects, standard decoding parameters have limited impact, and fine-tuning can shift scores in ways largely independent of the tuning data's political content, with results generalizable to 8 Values and across model sizes and quantization. The work underscores validity concerns for current bias benchmarks and motivates robust, mechanism-aware measures of political content encoding in LLMs.

Abstract

The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.

Paper Structure

This paper contains 16 sections, 3 figures, 29 tables.

Table of Contents

  1. Introduction
  2. Experiment Setup
  3. RQ1: Decoding Params & Prompting
  4. RQ2: Fine-Tuning
  5. Related Work
  6. Conclusion & Future Work
  7. Appendix
  8. Datasets
  9. Effect of Finetuning on PCT scores
  10. PCT Score Detailed Results
  11. Results for 8 Values Test
  12. Effect of Model Size & Quantization.
  13. Model shift analysis
  14. Experimental setup
  15. Evaluation results
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Example PCT scores in Mistral-7B-Instruct-v0.3 model before and after finetuning with multiple datasets with various generation parameters but the same prompt(prompt 9)\ref{['ssec:prompts']}. We systematically investigate the effect of these factors on these scores.
  • Figure 2: The change in PCT scores for different models in combination of finetuning (one dataset per model) and for randomly selected prompt and decoding parameters.
  • Figure 3: The PCT score changes for Gemma after finetuning on different datasets.