CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation
Nikola Ljubešić, Taja Kuzman
TL;DR
The paper introduces CLASSLA-web, a largest-ever, highly comparable collection of web corpora for seven South Slavic languages, built from MaCoCu crawls to ensure uniform coverage and post-processing. It combines state-of-the-art linguistic annotation via the CLASSLA-Stanza web module with multilingual genre labeling through the X-GENRE classifier, enabling cross-language comparisons of genre distribution and annotation quality. The resulting corpora total about 12.95B tokens from roughly 26.08M documents, with Macedonian represented by the first linguistically annotated general corpus for that language and Bulgarian constituting the largest subset; all data are publicly accessible through CLARIN.SI. The work demonstrates consistent genre distributions across languages, reveals significant correlations between genre mix and economic development, and provides a valuable resource for NLP model development (e.g., language models) and corpus-based linguistics in under-resourced South Slavic languages.
Abstract
This paper presents a collection of highly comparable web corpora of Slovenian, Croatian, Bosnian, Montenegrin, Serbian, Macedonian, and Bulgarian, covering thereby the whole spectrum of official languages in the South Slavic language space. The collection of these corpora comprises a total of 13 billion tokens of texts from 26 million documents. The comparability of the corpora is ensured by a comparable crawling setup and the usage of identical crawling and post-processing technology. All the corpora were linguistically annotated with the state-of-the-art CLASSLA-Stanza linguistic processing pipeline, and enriched with document-level genre information via the Transformer-based multilingual X-GENRE classifier, which further enhances comparability at the level of linguistic annotation and metadata enrichment. The genre-focused analysis of the resulting corpora shows a rather consistent distribution of genres throughout the seven corpora, with variations in the most prominent genre categories being well-explained by the economic strength of each language community. A comparison of the distribution of genre categories across the corpora indicates that web corpora from less developed countries primarily consist of news articles. Conversely, web corpora from economically more developed countries exhibit a smaller proportion of news content, with a greater presence of promotional and opinionated texts.