Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frame In, Frame Out: Do LLMs Generate More Biased News Headlines than Humans?

Valeria Pastorino, Nafise Sadat Moosavi

TL;DR

The paper investigates whether large language models (LLMs) generate more biased news headlines through framing than humans. It implements a prompting-based framing detector and a jury ensemble of LLMs to assess framing across 27 LM families on XSUM-derived headlines, including both out-of-the-box and fine-tuned variants. The findings show that LLMs generally exhibit more framing than human-authored headlines, with the highest biases in political and socially sensitive topics; fine-tuning on XSUM can mitigate framing to some extent, and there is notable cross-architecture variation. The study highlights the need for framing-aware evaluation benchmarks and post-training mitigation to ensure AI-generated news remains neutral and fairly represented, especially as LLM deployment widens across domains. Overall, the work provides a rigorous, topic-aware analysis of framing biases in modern LLMs and informs safer deployment practices for automated news generation.

Abstract

Framing in media critically shapes public perception by selectively emphasizing some details while downplaying others. With the rise of large language models in automated news and content creation, there is growing concern that these systems may introduce or even amplify framing biases compared to human authors. In this paper, we explore how framing manifests in both out-of-the-box and fine-tuned LLM-generated news content. Our analysis reveals that, particularly in politically and socially sensitive contexts, LLMs tend to exhibit more pronounced framing than their human counterparts. In addition, we observe significant variation in framing tendencies across different model architectures, with some models displaying notably higher biases. These findings point to the need for effective post-training mitigation strategies and tighter evaluation frameworks to ensure that automated news content upholds the standards of balanced reporting.

Frame In, Frame Out: Do LLMs Generate More Biased News Headlines than Humans?

TL;DR

The paper investigates whether large language models (LLMs) generate more biased news headlines through framing than humans. It implements a prompting-based framing detector and a jury ensemble of LLMs to assess framing across 27 LM families on XSUM-derived headlines, including both out-of-the-box and fine-tuned variants. The findings show that LLMs generally exhibit more framing than human-authored headlines, with the highest biases in political and socially sensitive topics; fine-tuning on XSUM can mitigate framing to some extent, and there is notable cross-architecture variation. The study highlights the need for framing-aware evaluation benchmarks and post-training mitigation to ensure AI-generated news remains neutral and fairly represented, especially as LLM deployment widens across domains. Overall, the work provides a rigorous, topic-aware analysis of framing biases in modern LLMs and informs safer deployment practices for automated news generation.

Abstract

Framing in media critically shapes public perception by selectively emphasizing some details while downplaying others. With the rise of large language models in automated news and content creation, there is growing concern that these systems may introduce or even amplify framing biases compared to human authors. In this paper, we explore how framing manifests in both out-of-the-box and fine-tuned LLM-generated news content. Our analysis reveals that, particularly in politically and socially sensitive contexts, LLMs tend to exhibit more pronounced framing than their human counterparts. In addition, we observe significant variation in framing tendencies across different model architectures, with some models displaying notably higher biases. These findings point to the need for effective post-training mitigation strategies and tighter evaluation frameworks to ensure that automated news content upholds the standards of balanced reporting.
Paper Structure (20 sections, 2 figures)

This paper contains 20 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Data
  5. Analysis
  6. Framing Bias: Human vs. LLMs
  7. Impact of Model Scale and Pretraining Data:
  8. Effect of Fine-Tuning:
  9. Size Variation Within Model Families:
  10. Framing and Topics Relationship
  11. Length vs. Framing Presence
  12. Conclusions
  13. Experimental Details
  14. Framing Detection Experiment
  15. Topics Analysis
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Framing tendencies of LLMs, measured as the percentage of framed content. Red and blue lines indicate human baselines for two data subsets.
  • Figure 2: Percentage of "Framed" outputs across models and topics.