Document Quality Scoring for Web Crawling

TL;DR

This work tackles the problem of abundant low-quality web content by introducing neural semantic quality scoring to guide crawling and indexing. It formalizes a quality estimator $\mathcal{Q}_\theta$ and a QOracle crawling priority, integrates a Docker-based scorer with the Resilipipe preprocessing pipeline, and evaluates on ClueWeb22-B and Open Web Search datasets using downstream recall metrics. Results show that prioritizing semantically high-quality pages improves early retrieval effectiveness, and high-quality pages tend to link to other high-quality pages, enabling effective approximate crawling when full text is not pre-downloaded. The authors provide reproducible software and data artifacts to facilitate adoption in production web search systems.

Abstract

The internet contains large amounts of low-quality content, yet users expect web search engines to deliver high-quality, relevant results. The abundant presence of low-quality pages can negatively impact retrieval and crawling processes by wasting resources on these documents. Therefore, search engines can greatly benefit from techniques that leverage efficient quality estimation methods to mitigate these negative impacts. Quality scoring methods for web pages are useful for many processes typical for web search systems, including static index pruning, index tiering, and crawling. Building on work by Chang et al.~\cite{chang2024neural}, who proposed using neural estimators of semantic quality for static index pruning, we extend their approach and apply their neural quality scorers to assess the semantic quality of web pages in crawling prioritisation tasks. In our experimental analysis, we found that prioritising semantically high-quality pages over low-quality ones can improve downstream search effectiveness. Our software contribution consists of a Docker container that computes an effective quality score for a given web page, allowing the quality scorer to be easily included and used in other components of web search systems.

Figures (6)

• Figure 1: Architecture of our containerised quality scorer module. The quality score is added to the input as an additional column. The input and output format do not have to be identical.
• Figure 2: Overview of the integration of our quality scoring module (highlighted in green) in the Resilipipe pre-processing pipeline, adapted from ows-book.
• Figure 3: Comparison between the distributions of the quality score computed for subsets of OWS (main, legal), and for ClueWeb22-B. All the histograms are generated using $15$ bins.
• Figure 4: Boxplot and histogram plot of the distribution of the quality scores of relevant and irrelevant web pages for $2867$ judged queries from the union of MSM-WS with RQ. However, in the histogram plot, to make the distributions comparable, we performed undersampling.
• Figure 5: Comparison between the downstream retrieval effectiveness of our QOracle crawler, and baseline BFS and DFS crawlers in terms of $R@100$ computed on a mixed query set composed of $850$ judged queries from RQ, and $850$ judged queries from MSM-WS. Hollow markers denote statistically significant differences w.r.t. the two baselines, whereas filled markers denote differences that are not statistically significant.