Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

Max Dallabetta, Conrad Dobberstein, Adrian Breiding, Alan Akbik

TL;DR

The evaluation shows that Fundus yields significantly higher quality extractions than prior work, and this paper gives an overview of the framework, discusses the design choices, and presents a comparative evaluation against other popular news scrapers.

Abstract

This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work. The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.

Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

TL;DR

The evaluation shows that Fundus yields significantly higher quality extractions than prior work, and this paper gives an overview of the framework, discusses the design choices, and presents a comparative evaluation against other popular news scrapers.

Abstract

This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work. The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.
Paper Structure (22 sections, 4 figures, 6 tables)

This paper contains 22 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction and Motivation
  2. Related Work
  3. Content Extraction
  4. Crawling
  5. Fundus
  6. Usage Example
  7. Articles
  8. Publishers and Collections
  9. Forward and Backward Crawling
  10. Content Extraction
  11. Evaluation
  12. Experimental Setup
  13. Evaluation Dataset
  14. Evaluation Metric
  15. Results and Discussion
  16. ...and 7 more sections

Figures (4)

  • Figure 1: An example article scraped by Fundus. Next to the plain text of the article, attributes such as title, authors, paragraphs, subheadlines and topics are directly accessible.
  • Figure 2: Two example usages of Fundus to crawl articles from (1) all supported US-based publishers, and (2) only one specific German publisher ("Deutsche Welle").
  • Figure 3: Distribution of ROUGE-LSum F1-scores of scraper extractions. The scrapers are sorted in descending order over the F1-score.
  • Figure 4: An example usage of Fundus to crawl articles from CC-NEWS