Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Left, Right, or Center? Evaluating LLM Framing in News Classification and Generation

Molly Kennedy, Ali Parker, Yihong Liu, Hinrich Schütze

TL;DR

The paper investigates whether LLMs exhibit ideological framing in two contexts: classifying news with left/center/right labels and generating perspective-conditioned summaries. Using a fixed AllSides reference and a two-stage setup across nine models, it reveals a pronounced center-defaulting tendency in both classification and generation. Grok 4 emerges as the most ideologically expressive generator, while Claude Sonnet 4.5 and Llama 3.1 demonstrate stronger bias-rating performance among commercial and open-weight models, respectively. The findings highlight safety-alignment decisions as a key driver of framing behavior, underscoring important tradeoffs between guardrails and the ability to convey ideological nuance in journalism-like tasks. Overall, the work provides a practical, model-agnostic framework for diagnosing framing in LLMs and emphasizes the need for careful design when deploying LLMs in news-generation workflows.

Abstract

Large Language Model (LLM) based summarization and text generation are increasingly used for producing and rewriting text, raising concerns about political framing in journalism where subtle wording choices can shape interpretation. Across nine state-of-the-art LLMs, we study political framing by testing whether LLMs' classification-based bias signals align with framing behavior in their generated summaries. We first compare few-shot ideology predictions against LEFT/CENTER/RIGHT labels. We then generate "steered" summaries under FAITHFUL, CENTRIST, LEFT, and RIGHT prompts, and score all outputs using a single fixed ideology evaluator. We find pervasive ideological center-collapse in both article-level ratings and generated text, indicating a systematic tendency toward centrist framing. Among evaluated models, Grok 4 is by far the most ideologically expressive generator, while Claude Sonnet 4.5 and Llama 3.1 achieve the strongest bias-rating performance among commercial and open-weight models, respectively.

Left, Right, or Center? Evaluating LLM Framing in News Classification and Generation

TL;DR

The paper investigates whether LLMs exhibit ideological framing in two contexts: classifying news with left/center/right labels and generating perspective-conditioned summaries. Using a fixed AllSides reference and a two-stage setup across nine models, it reveals a pronounced center-defaulting tendency in both classification and generation. Grok 4 emerges as the most ideologically expressive generator, while Claude Sonnet 4.5 and Llama 3.1 demonstrate stronger bias-rating performance among commercial and open-weight models, respectively. The findings highlight safety-alignment decisions as a key driver of framing behavior, underscoring important tradeoffs between guardrails and the ability to convey ideological nuance in journalism-like tasks. Overall, the work provides a practical, model-agnostic framework for diagnosing framing in LLMs and emphasizes the need for careful design when deploying LLMs in news-generation workflows.

Abstract

Large Language Model (LLM) based summarization and text generation are increasingly used for producing and rewriting text, raising concerns about political framing in journalism where subtle wording choices can shape interpretation. Across nine state-of-the-art LLMs, we study political framing by testing whether LLMs' classification-based bias signals align with framing behavior in their generated summaries. We first compare few-shot ideology predictions against LEFT/CENTER/RIGHT labels. We then generate "steered" summaries under FAITHFUL, CENTRIST, LEFT, and RIGHT prompts, and score all outputs using a single fixed ideology evaluator. We find pervasive ideological center-collapse in both article-level ratings and generated text, indicating a systematic tendency toward centrist framing. Among evaluated models, Grok 4 is by far the most ideologically expressive generator, while Claude Sonnet 4.5 and Llama 3.1 achieve the strongest bias-rating performance among commercial and open-weight models, respectively.
Paper Structure (25 sections, 2 figures, 3 tables)

This paper contains 25 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Media bias and ideology in news.
  4. Political bias and framing in LLMs.
  5. Steerability and controllable generation.
  6. Dataset
  7. Methodology
  8. Data.
  9. Stage 1: Political rating alignment analysis via Bias Classification
  10. Stage 2: Alignment under ideologically steered summary generation
  11. Ideology evaluation.
  12. Experiment Results
  13. Stage 1: political rating alignment performance
  14. Granular insight into the models' bias rating calibration
  15. Inter-model agreement elucidates common alignment across models.
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Confusion matrices for LLM political direction rating capability on 12k (balanced) political news articles.
  • Figure 2: Cohen's kappa score heatmap measuring the agreeability between each of the model's ratings as well as with the ground truth.