Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cleaner Pretraining Corpus Curation with Neural Web Scraping

Zhipeng Xu, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Ge Yu, Chenyan Xiong

TL;DR

NeuScraper introduces a neural approach to extract primary content from web pages by leveraging DOM-derived textual sequences and a lightweight transformer architecture. It demonstrates over 20% gains over traditional baselines in primary-content extraction on ClueWeb22 and shows that the cleaned data improves downstream language-model pretraining, including lower perplexity on target corpora. The method emphasizes speed on GPU and offers CPU-friendly quantization via onnxruntime, supporting scalable large-scale corpus creation. Overall, NeuScraper provides a data-driven solution to reduce noise in web-derived pretraining data and facilitate higher-quality language model training at scale.

Abstract

The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.

Cleaner Pretraining Corpus Curation with Neural Web Scraping

TL;DR

NeuScraper introduces a neural approach to extract primary content from web pages by leveraging DOM-derived textual sequences and a lightweight transformer architecture. It demonstrates over 20% gains over traditional baselines in primary-content extraction on ClueWeb22 and shows that the cleaned data improves downstream language-model pretraining, including lower perplexity on target corpora. The method emphasizes speed on GPU and offers CPU-friendly quantization via onnxruntime, supporting scalable large-scale corpus creation. Overall, NeuScraper provides a data-driven solution to reduce noise in web-derived pretraining data and facilitate higher-quality language model training at scale.

Abstract

The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.
Paper Structure (18 sections, 4 equations, 4 figures, 4 tables)

This paper contains 18 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Web Scraper
  4. Textual Sequence Modeling of Webpages
  5. Web Scraping with the Neural Method
  6. Experimental Methodology
  7. Evaluation Result
  8. Overall Performance
  9. Effectiveness of the Cleaned Web Data in Language Model Pretraining
  10. Evaluation on the Quality of Extracted Data Using NeuScraper
  11. Model Quantization for NeuScraper
  12. Conlusion
  13. Appendix
  14. License
  15. More Experimental Details of Overall Evaluation
  16. ...and 3 more sections

Figures (4)

  • Figure 1: The Pipeline of Primary Content Extraction Using NeuScraper (Neural Web Scraper).
  • Figure 2: The Effectiveness of Language Models Trained on Web Data to Reproduce the Target Corpora. Lower perplexity indicates more proficiency in language models for reproducing.
  • Figure 3: Case#1 of the Primary Content Extraction Results Using Different Scrapers. The extracted parts are highlighted with red boxes.
  • Figure 4: Case#2 of the Primary Content Extraction Results Using Different Scrapers. The extracted parts are highlighted with blue boxes.