Beyond the Surface: Probing the Ideological Depth of Large Language Models

TL;DR

The paper investigates how to quantify ideological depth in LLMs by combining steerability tests with an unsupervised analysis of internal political features. Using two open models, Llama-3.1-8B-Instruct and Gemma-2-9B-IT, it benchmarks prompt-based and activation steering across 12 topics and analyzes features with Sparse Autoencoders (SAEs). The findings show Gemma harbors a richer, more separable political feature space and higher steerability, while Llama exhibits brittleness and more refusals under conservative prompts; ablation of salient political features in Gemma increases refusals, providing causal evidence linking feature richness to instruction following. These results imply ideological depth is a measurable property with implications for AI alignment and monitoring, suggesting future work on mechanistic origins, real-time feature monitoring, and training interventions to shape ideological reasoning while mitigating risks.

Abstract

Large language models (LLMs) display recognizable political leanings, yet they vary significantly in their ability to represent a political orientation consistently. In this paper, we define ideological depth as (i) a model's ability to follow political instructions without failure (steerability), and (ii) the feature richness of its internal political representations measured with sparse autoencoders (SAEs), an unsupervised sparse dictionary learning (SDL) approach. Using Llama-3.1-8B-Instruct and Gemma-2-9B-IT as candidates, we compare prompt-based and activation-steering interventions and probe political features with publicly available SAEs. We find large, systematic differences: Gemma is more steerable in both directions and activates approximately 7.3x more distinct political features than Llama. Furthermore, causal ablations of a small targeted set of Gemma's political features to create a similar feature-poor setting induce consistent shifts in its behavior, with increased rates of refusals across topics. Together, these results indicate that refusals on benign political instructions or prompts can arise from capability deficits rather than safety guardrails. Ideological depth thus emerges as a measurable property of LLMs, and steerability serves as a window into their latent political architecture.

Figures (14)

• Figure 1: Topic-wise refusal (“null”) rates. Llama refuses more often on socially contentious issues, while Gemma remains responsive.
• Figure 2: Estimated ideological positions across nine prompting conditions. Variance increases under conservative personas or arguments, revealing unstable conservative representations.
• Figure 3: Effects of Contrastive Activation Addition (CAA) and Steering Target Atoms (STA) on response polarity. Gemma shifts smoothly toward conservative answers; Llama mostly increases refusals. Lines show the mean of 5 trials; shaded regions represent the 95% confidence interval.
• Figure 4: Ideal-point estimates under activation steering with multiplier +1 and -1 corresponding to liberal and conservative steering, respectively. Gemma forms distinct clusters for liberal vs. conservative directions; Llama’s overlap indicates weaker separation.
• Figure 5: Distributions of feature evaluation scores. Gemma’s SAE features achieve higher thematic coherence (right) and stronger predictive validity than Llama’s (left).