When Neutral Summaries are not that Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries

TL;DR

This work quantifies political neutrality in LLM-generated news summaries by prompting four open-source models to produce neutral, Democrat-aligned, and Republican-aligned outputs across five US political issues for 20,344 articles, and by introducing a Polarization Index $\mathcal{P}$ and token-divergence metrics. It demonstrates a consistent pro-Democratic tilt in several models, with the strongest bias observed on Gun Control/Rights and Healthcare, and reveals algorithmic monoculture through vocabulary overlaps and cross-model transferability of biases quantified by $\mathcal{CI}$ and $\mathcal{A}$. The paper provides a tri-perspective, quantitative framework for neutrality assessment and releases a large tri-perspective dataset to support future research in political alignment and AI fairness. These findings highlight the potential risks of AI-assisted summarization to shape political perception and call for methodological and policy safeguards to preserve neutral information ecosystems in politically salient domains.

Abstract

In an era where societal narratives are increasingly shaped by algorithmic curation, investigating the political neutrality of LLMs is an important research question. This study presents a fresh perspective on quantifying the political neutrality of LLMs through the lens of abstractive text summarization of polarizing news articles. We consider five pressing issues in current US politics: abortion, gun control/rights, healthcare, immigration, and LGBTQ+ rights. Via a substantial corpus of 20,344 news articles, our study reveals a consistent trend towards pro-Democratic biases in several well-known LLMs, with gun control and healthcare exhibiting the most pronounced biases (max polarization differences of -9.49% and -6.14%, respectively). Further analysis uncovers a strong convergence in the vocabulary of the LLM outputs for these divisive topics (55% overlap for Democrat-leaning representations, 52% for Republican). Being months away from a US election of consequence, we consider our findings important.

Figures (3)

• Figure 1: This image depicts the publication years of articles scraped for our study, spanning from 2019 to 2024. Each bar represents the quantity of articles from a specific year included in the analysis.
• Figure 2: Heatmaps illustrating the consistency index $\mathcal{CI}$ across various large language models (LLMs) for both Democrat and Republican-leaning vocabularies. These visualizations serve as a measure of ideological congruence, revealing patterns of uniformity, if any, in how different LLMs linguistically frame their opinions.
• Figure 3: Comparative heatmaps illustrating classifier performance ($\mathcal{A}$ scores) in identifying political biases within LLM-generated summaries. These visualizations reveal the degree of transferability of ideological biases and linguistic strategies, providing insights into the prevalence of algorithmic monoculture.