Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora

Taja Kuzman Pungeršek, Peter Rupnik, Vít Suchomel, Nikola Ljubešić

TL;DR

The paper demonstrates that iterative crawling of national top-level domains yields rapidly expanding, high-coverage web corpora for seven South Slavic languages, embodied in CLASSLA-web 2.0 (17.0B words, 38.1M texts) with automatic genre and topic annotations. It compares 2.0 to the prior 1.0 release, finding about 80% content is unique to 2.0 over a two-year interval and highlighting substantial web-content turnover. The authors detail the CLASSLA-web construction pipeline, including crawling, language identification, post-processing, and multilevel annotations, plus quality-control steps that reveal a growing share of machine-generated content requiring manual domain validation. The work provides a valuable, openly accessible resource for NLP and linguistics, and argues for ongoing, biannual crawling to keep pace with rapid web evolution while maintaining data quality.

Abstract

Crawling national top-level domains has proven to be highly effective for collecting texts in less-resourced languages. This approach has been recently used for South Slavic languages and resulted in the largest general corpora for this language group: the CLASSLA-web 1.0 corpora. Building on this success, we established a continuous crawling infrastructure for iterative national top-level domain crawling across South Slavic and related webs. We present the first outcome of this crawling infrastructure - the CLASSLA-web 2.0 corpus collection, with substantially larger web corpora containing 17.0 billion words in 38.1 million texts in seven languages: Bosnian, Bulgarian, Croatian, Macedonian, Montenegrin, Serbian, and Slovenian. In addition to genre categories, the new version is also automatically annotated with topic labels. Comparing CLASSLA-web 2.0 with its predecessor reveals that only one-fifth of the texts overlap, showing that re-crawling after just two years yields largely new content. However, while the new web crawls bring growing gains, we also notice growing pains - a manual inspection of top domains reveals a visible degradation of web content, as machine-generated sites now contribute a significant portion of texts.

The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora

TL;DR

The paper demonstrates that iterative crawling of national top-level domains yields rapidly expanding, high-coverage web corpora for seven South Slavic languages, embodied in CLASSLA-web 2.0 (17.0B words, 38.1M texts) with automatic genre and topic annotations. It compares 2.0 to the prior 1.0 release, finding about 80% content is unique to 2.0 over a two-year interval and highlighting substantial web-content turnover. The authors detail the CLASSLA-web construction pipeline, including crawling, language identification, post-processing, and multilevel annotations, plus quality-control steps that reveal a growing share of machine-generated content requiring manual domain validation. The work provides a valuable, openly accessible resource for NLP and linguistics, and argues for ongoing, biannual crawling to keep pace with rapid web evolution while maintaining data quality.

Abstract

Crawling national top-level domains has proven to be highly effective for collecting texts in less-resourced languages. This approach has been recently used for South Slavic languages and resulted in the largest general corpora for this language group: the CLASSLA-web 1.0 corpora. Building on this success, we established a continuous crawling infrastructure for iterative national top-level domain crawling across South Slavic and related webs. We present the first outcome of this crawling infrastructure - the CLASSLA-web 2.0 corpus collection, with substantially larger web corpora containing 17.0 billion words in 38.1 million texts in seven languages: Bosnian, Bulgarian, Croatian, Macedonian, Montenegrin, Serbian, and Slovenian. In addition to genre categories, the new version is also automatically annotated with topic labels. Comparing CLASSLA-web 2.0 with its predecessor reveals that only one-fifth of the texts overlap, showing that re-crawling after just two years yields largely new content. However, while the new web crawls bring growing gains, we also notice growing pains - a manual inspection of top domains reveals a visible degradation of web content, as machine-generated sites now contribute a significant portion of texts.
Paper Structure (22 sections, 1 equation, 7 figures, 2 tables)

This paper contains 22 sections, 1 equation, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CLASSLA-web Construction Pipeline
  4. Web Crawling Pipeline
  5. Language Identification
  6. Post-Processing
  7. Manual Inspection of Top Domains
  8. Genre Annotation
  9. Topic Annotation
  10. Linguistic Annotation
  11. Differences in the Pipeline Between Versions 1.0 and 2.0
  12. CLASSLA-web 2.0 Corpora
  13. Genre and Topic Distribution
  14. Evolution of the Web between CLASSLA-web 1.0 and CLASSLA-web 2.0
  15. Size Gains
  16. ...and 7 more sections

Figures (7)

  • Figure 1: The CLASSLA-web construction pipeline (detailed in Section \ref{['sec:pipeline']}), consisting of several key steps: web crawling, language identification using multiple tools, post-processing to ensure high-quality corpus content, automatic genre and topic annotation, and linguistic annotation.
  • Figure 2: Distribution of genre categories across South Slavic CLASSLA-web 2.0 corpora.
  • Figure 3: Most frequent topics in News texts in CLASSLA-web 2.0 corpora. The topics that are represented by 10% or more texts in at least one corpus are included in the figure.
  • Figure 4: Comparison of sizes in millions of words between CLASSLA-web 1.0 and CLASSLA-web 2.0 corpora for Bosnian (bs), Bulgarian (bg), Croatian (hr), Macedonian (mk), Montenegrin (cnr), Serbian (sr) and Slovenian (sl) corpora.
  • Figure 5: Number of texts that are unique in CLASSLA-web 1.0 or 2.0 versions, number of texts that are shared between the two versions, and a total number when the two corpora are merged.
  • ...and 2 more figures