ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information

TL;DR

ClueWeb22 delivers a real-distribution, 10 billion-page web corpus with industry-grade signals beyond raw HTML, including browser-rendered visuals, vDOM-based semantics, and annotated content fields. The dataset is sampled from a commercial search index using predicted click likelihood across three distribution categories (B/A/L) to reflect real web traffic while balancing quality and cost. A plug-and-play content extraction pipeline (Visual Render, Semantic Annotator, Enriched Parser) plus an Anchor Graph provides ready-to-use clean text, structured content, and link signals, enabling robust research for IR, retrieval-augmented NLP, and large-scale pretraining. Compared with CommonCrawl, ClueWeb22 prioritizes distribution fidelity and signal richness, while remaining openly licensed to democratize access for the research community.

Abstract

ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high quality, large scale web corpus to support a range of academic and industry research, for example, in information systems, retrieval-augmented AI systems, and model pretraining. Compared with earlier ClueWeb corpora, the ClueWeb22 corpus is larger, more varied, of higher-quality, and aligned with the document distributions in commercial web search. Besides raw HTML, ClueWeb22 includes rich information about the web pages provided by industry-standard document understanding systems, including the visual representation of pages rendered by a web browser, parsed HTML structure information from a neural network parser, and pre-processed cleaned document text to lower the barrier to entry. Many of these signals have been widely used in industry but are available to the research community for the first time at this scale.

Figures (7)

• Figure 1: The system pipeline to extract content from ClueWeb22 web pages. The extracted Clean Content is available for all ClueWeb22 pages which is a good starting point for common applications. The Annotated vDOM is available in ClueWeb22-A to support customized content extraction for more specific needs.
• Figure 2: The distributions of the top twenty URL domains in each part of the ClueWeb22 dataset. The top twenty domains cover 12.6% (B), 12.2% (A), and 13.3 % (L) of entire corpus.
• Figure 3: The distributions of the top twenty languages in ClueWeb22. The top twenty language cover 97.2% (B), 96.3% (A), and 95.0 % (L) of the full set.
• Figure 4: The distribution of the top twenty topics in ClueWeb22, classified by a standard text classifier. The top twenty topics cover 12.0% (B), 12.1% (A), and 12.7 % (L) of the full set.
• Figure 5: The length distribution of HTML files of ClueWeb22 counted as the number of raw HTML characters (code and text), characters of all texts, and tokens of all texts. The mean is shown in the parentheses.