Building Retrieval Systems for the ClueWeb22-B Corpus

TL;DR

This work establishes retrieval baselines for the English portion of ClueWeb22-B by implementing and evaluating three components: Lucene-based BM25 first-stage retrieval, dense vector retrieval using Contriever with Faiss indexing, and neural rerankers trained on MS-MARCO and ClueWeb-derived data. The study demonstrates strong BM25 recall in the first stage, but denser semantic retrieval lags in recall, and anchor-text–driven rerankers can degrade performance, with MS-MARCO pretraining after anchor-data yielding the best gains among reranker configurations. The authors provide a public-facing API service on a CMU cluster and discuss practical trade-offs, such as latency and data biases, to guide future benchmarking and model development. Overall, the paper delivers a replicable baseline suite that enables fair comparison of retrieval models on a large-scale web corpus and highlights important considerations for training data and evaluation in retrieval systems.

Abstract

The ClueWeb22 dataset containing nearly 10 billion documents was released in 2022 to support academic and industry research. The goal of this project was to build retrieval baselines for the English section of the "super head" part (category B) of this dataset. These baselines can then be used by the research community to compare their systems and also to generate data to train/evaluate new retrieval and ranking algorithms. The report covers sparse and dense first stage retrievals as well as neural rerankers that were implemented for this dataset. These systems are available as a service on a Carnegie Mellon University cluster.

Figures (1)

• Figure 1: Train loss for the reranker trained on anchor text