Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
Nicolas Steinacker-Olsztyn, Devashish Gosain, Ha Dao
TL;DR
The paper addresses how robots.txt gatekeeping differs between reputable news sites and misinformation sites and what this means for AI training data. It combines credibility-labeled website lists, AI agent catalogs, multi-geo measurements, and Internet Archive snapshots to assess AI crawler directives and blocking behavior. The findings show a substantial and growing gap: about 60% of reputable sites disallow at least one AI agent, versus roughly 9% of misinformation sites, with reputable sites also more likely to explicitly list many AI agents and to employ active blocking. These results have implications for data ethics, web transparency, and the future of LLM training, suggesting that access controls may increasingly shape the content incorporated into AI models.
Abstract
Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark contrast: 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files. Reputable sites forbid an average of 15.5 AI user agents, while misinformation sites prohibit fewer than one. We then measure active blocking behavior, where websites refuse to return content when HTTP requests include AI crawler user agents, and reveal that both categories of websites utilize it. Notably, the behavior of reputable news websites in this regard aligns more closely with their declared robots.txt directive than that of misinformation websites. Finally, our longitudinal analysis reveals that this gap has widened over time, with AI-blocking by reputable sites rising from 23% in September 2023 to nearly 60% by May 2025. Our findings highlight a growing asymmetry in content accessibility that may shape the training data available to LLMs, raising essential questions for web transparency, data ethics, and the future of AI training practices.