Automatic Generation of Web Censorship Probe Lists

TL;DR

This work presents an automated pipeline to generate up-to-date domain probe lists for web censorship measurement, addressing biases and scalability in manual/crowdsourced approaches. Starting from a large seed of existing URLs, the pipeline performs language detection, multilingual text processing, and multi-method topic/keyword extraction (BERTopic, LDA, Top2Vec), then expands topics via ChatGPT prompts and Google Trends, and finally crawls Google Search results to assemble a 119,255-URL probe list. Systematic testing across 11 global locations over four months, supplemented by OONI Probe validation, reveals hundreds of previously unseen domains that may be blocked, with particular emphasis on China where DNS and IP-based blocking are evident. Compared with prior efforts, the approach yields significantly more potential censorship signals and demonstrates the practicality of continuous, automated probe-list updating to support scalable censorship measurement. The methodology offers a foundation for integrating automated probe lists with global measurement platforms while highlighting regional biases and ethical considerations inherent in censorship research.

Abstract

Domain probe lists--used to determine which URLs to probe for Web censorship--play a critical role in Internet censorship measurement studies. Indeed, the size and accuracy of the domain probe list limits the set of censored pages that can be detected; inaccurate lists can lead to an incomplete view of the censorship landscape or biased results. Previous efforts to generate domain probe lists have been mostly manual or crowdsourced. This approach is time-consuming, prone to errors, and does not scale well to the ever-changing censorship landscape. In this paper, we explore methods for automatically generating probe lists that are both comprehensive and up-to-date for Web censorship measurement. We start from an initial set of 139,957 unique URLs from various existing test lists consisting of pages from a variety of languages to generate new candidate pages. By analyzing content from these URLs (i.e., performing topic and keyword extraction), expanding these topics, and using them as a feed to search engines, our method produces 119,255 new URLs across 35,147 domains. We then test the new candidate pages by attempting to access each URL from servers in eleven different global locations over a span of four months to check for their connectivity and potential signs of censorship. Our measurements reveal that our method discovered over 1,400 domains--not present in the original dataset--we suspect to be blocked. In short, automatically updating probe lists is possible, and can help further automate censorship measurements at scale.

Figures (4)

• Figure 1: Flowchart of the URL generation process. The dashed boxes correspond to different subsections in the body of the text.
• Figure 2: The frequency (CDF) of percentage of consistent runs of URLs for each location outside the baseline. The vertical line (at 95%) is the cutoff for considering responses from a URL to be inconsistent and excluding it from our data.
• Figure 3: Jaccard similarity index of inaccessible domains between locations.
• Figure 4: Jaccard similarity index of domains that returned HTTP response code errors between locations.