Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Impact of LLMs on Online News Consumption and Production

Hangcheng Zhao, Ron Berman

TL;DR

This paper investigates how large language models and GenAI tools reshape online news consumption and production, focusing on publisher traffic, access-control responses, labor demand, and content formats. It builds a high-frequency publisher panel combining traffic data (SimilarWeb, Comscore), robots.txt/page-structure (HTTP Archive), content proxies (Wayback), and hiring data (Revelio) to identify causal effects via change-point analysis, blocked crawler differences, and synthetic/DiD methods. Key findings show that publisher traffic declines emerge after August 2024, blocking GenAI crawlers reduces both total and human traffic for large publishers, newsroom hiring does not contract in the near term, and publishers shift toward richer, interactive content rather than scaling text output. These results imply GenAI is not yet a full substitute for traditional news production, and publishers' adaptive strategies—access control, hiring composition, and content-format changes—carry unintended consequences and important implications for policy and industry practice.

Abstract

Large language models (LLMs) change how consumers acquire information online; their bots also crawl news publishers' websites for training data and to answer consumer queries; and they provide tools that can lower the cost of content creation. These changes lead to predictions of adverse impact on news publishers in the form of lowered consumer demand, reduced demand for newsroom employees, and an increase in news "slop." Consequently, some publishers strategically responded by blocking LLM access to their websites using the robots.txt file standard. Using high-frequency granular data, we document four effects related to the predicted shifts in news publishing following the introduction of generative AI (GenAI). First, we find a consistent and moderate decline in traffic to news publishers occurring after August 2024. Second, using a difference-in-differences approach, we find that blocking GenAI bots can have adverse effects on large publishers by reducing total website traffic by 23% and real consumer traffic by 14% compared to not blocking. Third, on the hiring side, we do not find evidence that LLMs are replacing editorial or content-production jobs yet. The share of new editorial and content-production job listings increases over time. Fourth, regarding content production, we find no evidence that large publishers increased text volume; instead, they significantly increased rich content and use more advertising and targeting technologies. Together, these findings provide early evidence of some unforeseen impacts of the introduction of LLMs on news production and consumption.

The Impact of LLMs on Online News Consumption and Production

TL;DR

This paper investigates how large language models and GenAI tools reshape online news consumption and production, focusing on publisher traffic, access-control responses, labor demand, and content formats. It builds a high-frequency publisher panel combining traffic data (SimilarWeb, Comscore), robots.txt/page-structure (HTTP Archive), content proxies (Wayback), and hiring data (Revelio) to identify causal effects via change-point analysis, blocked crawler differences, and synthetic/DiD methods. Key findings show that publisher traffic declines emerge after August 2024, blocking GenAI crawlers reduces both total and human traffic for large publishers, newsroom hiring does not contract in the near term, and publishers shift toward richer, interactive content rather than scaling text output. These results imply GenAI is not yet a full substitute for traditional news production, and publishers' adaptive strategies—access control, hiring composition, and content-format changes—carry unintended consequences and important implications for policy and industry practice.

Abstract

Large language models (LLMs) change how consumers acquire information online; their bots also crawl news publishers' websites for training data and to answer consumer queries; and they provide tools that can lower the cost of content creation. These changes lead to predictions of adverse impact on news publishers in the form of lowered consumer demand, reduced demand for newsroom employees, and an increase in news "slop." Consequently, some publishers strategically responded by blocking LLM access to their websites using the robots.txt file standard. Using high-frequency granular data, we document four effects related to the predicted shifts in news publishing following the introduction of generative AI (GenAI). First, we find a consistent and moderate decline in traffic to news publishers occurring after August 2024. Second, using a difference-in-differences approach, we find that blocking GenAI bots can have adverse effects on large publishers by reducing total website traffic by 23% and real consumer traffic by 14% compared to not blocking. Third, on the hiring side, we do not find evidence that LLMs are replacing editorial or content-production jobs yet. The share of new editorial and content-production job listings increases over time. Fourth, regarding content production, we find no evidence that large publishers increased text volume; instead, they significantly increased rich content and use more advertising and targeting technologies. Together, these findings provide early evidence of some unforeseen impacts of the introduction of LLMs on news production and consumption.
Paper Structure (21 sections, 3 equations, 17 figures, 12 tables)

This paper contains 21 sections, 3 equations, 17 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Publisher traffic declines, but the decline emerges only after August 2024.
  3. Some publishers decide to block LLM traffic, but blocking reduces both total and human traffic for large publishers.
  4. No near-term contraction in newsroom hiring relative to other roles.
  5. Publishers do not scale up textual production; they shift toward richer pages and embedded components.
  6. Contribution and Related Work
  7. Data
  8. Website traffic
  9. SimilarWeb
  10. Comscore Web-Behavior Panel
  11. Publisher characteristics and content quantity
  12. Robots.txt and page structure
  13. Content quantity proxies.
  14. Job postings and employment
  15. Sample construction and merge across sources
  16. ...and 6 more sections

Figures (17)

  • Figure 1: Publishers' Daily Traffic Trend
  • Figure 2: Change-Point Detection for Daily Traffic
  • Figure 3: Fraction of websites that disallow GenAI bots
  • Figure 4: Staggered DiD of blocking GenAI bots on publisher traffic.
  • Figure 5: Staggered DiD estimates for Comscore traffic by publisher size group
  • ...and 12 more figures