Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wikipedia in the Era of LLMs: Evolution and Risks

Siming Huang, Yuliang Xu, Mingmeng Geng, Yao Wan, Dongping Chen

TL;DR

This work investigates how Large Language Models influence Wikipedia and the downstream NLP tasks that rely on its content. It combines data analysis of page views, word usage, and linguistic style with simulations of LLM revision to quantify direct impacts, finding modest but observable shifts (often 1–2%) in some categories. It further examines indirect effects on machine translation benchmarks and Retrieval-Augmented Generation (RAG), showing that LLM-influenced content can inflate translation scores and reduce RAG accuracy, especially for recent events or when revisions alter key information. The results emphasize the bidirectional coevolution of Wikipedia and LLMs, underscoring the need for careful risk assessment and robust methods to detect and mitigate content contamination in large-scale knowledge sources used by AI systems.

Abstract

In this paper, we present a thorough analysis of the impact of Large Language Models (LLMs) on Wikipedia, examining the evolution of Wikipedia through existing data and using simulations to explore potential risks. We begin by analyzing page views and article content to study Wikipedia's recent changes and assess the impact of LLMs. Subsequently, we evaluate how LLMs affect various Natural Language Processing (NLP) tasks related to Wikipedia, including machine translation and retrieval-augmented generation (RAG). Our findings and simulation results reveal that Wikipedia articles have been influenced by LLMs, with an impact of approximately 1%-2% in certain categories. If the machine translation benchmark based on Wikipedia is influenced by LLMs, the scores of the models may become inflated, and the comparative results among models might shift as well. Moreover, the effectiveness of RAG might decrease if the knowledge base becomes polluted by LLM-generated content. While LLMs have not yet fully changed Wikipedia's language and knowledge structures, we believe that our empirical findings signal the need for careful consideration of potential future risks.

Wikipedia in the Era of LLMs: Evolution and Risks

TL;DR

This work investigates how Large Language Models influence Wikipedia and the downstream NLP tasks that rely on its content. It combines data analysis of page views, word usage, and linguistic style with simulations of LLM revision to quantify direct impacts, finding modest but observable shifts (often 1–2%) in some categories. It further examines indirect effects on machine translation benchmarks and Retrieval-Augmented Generation (RAG), showing that LLM-influenced content can inflate translation scores and reduce RAG accuracy, especially for recent events or when revisions alter key information. The results emphasize the bidirectional coevolution of Wikipedia and LLMs, underscoring the need for careful risk assessment and robust methods to detect and mitigate content contamination in large-scale knowledge sources used by AI systems.

Abstract

In this paper, we present a thorough analysis of the impact of Large Language Models (LLMs) on Wikipedia, examining the evolution of Wikipedia through existing data and using simulations to explore potential risks. We begin by analyzing page views and article content to study Wikipedia's recent changes and assess the impact of LLMs. Subsequently, we evaluate how LLMs affect various Natural Language Processing (NLP) tasks related to Wikipedia, including machine translation and retrieval-augmented generation (RAG). Our findings and simulation results reveal that Wikipedia articles have been influenced by LLMs, with an impact of approximately 1%-2% in certain categories. If the machine translation benchmark based on Wikipedia is influenced by LLMs, the scores of the models may become inflated, and the comparative results among models might shift as well. Moreover, the effectiveness of RAG might decrease if the knowledge base becomes polluted by LLM-generated content. While LLMs have not yet fully changed Wikipedia's language and knowledge structures, we believe that our empirical findings signal the need for careful consideration of potential future risks.

Paper Structure

This paper contains 66 sections, 1 equation, 27 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Wikipedia for NLP.
  4. LLMs for Wikipedia.
  5. Wikipedia.
  6. Estimation of LLM Impact.
  7. Data Collection
  8. Direct Impact from LLMs
  9. Direct Impact 1: Page View
  10. Finding 1:
  11. Direct Impact 2: Word Frequency
  12. Finding 2:
  13. Direct Impact 3: Linguistic Style
  14. Overall.
  15. Experiment Setups
  16. ...and 51 more sections

Figures (27)

  • Figure 1: Our work analyze the direct impact of LLMs on Wikipedia, and exploring the indirect impact of LLMs generated content on Wikipedia: Have LLMs already impacted Wikipedia, and if so, how might they influence the broader NLP community?
  • Figure 2: Monthly page views across different Wikipedia categories. The vertical axis represents the transformed page view values, standardized using the Inverse Hyperbolic Sine (IHS) function.
  • Figure 3: Word frequency in the first section of the Wikipedia articles.
  • Figure 4: LLM Impact: Estimated based on simulations of the first section of Featured Articles, using different word combinations across different categories of Wikipedia pages.
  • Figure 5: The results of linguistic style comparison, including the real Wikipedia pages and LLM-simulated pages. The three subplots below represent the differences compared to the data from 2020.
  • ...and 22 more figures