SWEb: A Large Web Dataset for the Scandinavian Languages

TL;DR

SWEb addresses the scarcity of large-scale open pretraining data for Scandinavian languages by introducing a model-based HTML content extractor and assembling a 1.01 trillion-token dataset from 98 Common Crawl snapshots. The pipeline uses a Markdown-based text output with a line-level transformer extractor, trained on modest annotated data, and applies lightweight quality filters and per-snapshot deduplication. HP-MEK benchmarking shows models trained on SWEb achieve competitive performance relative to FineWeb while leveraging substantially more high-quality tokens. By open-sourcing the data, code, and extractor, the work lowers barriers to building Scandinavian language models and provides a data-efficient blueprint for similar low-resource languages.

Abstract

This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce a new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare models trained on the SWEb data to models trained on FineWeb, with competitive results. All data, models and code are shared openly.

Figures (15)

• Figure 1: The SWEb pipeline. We use Common Crawl's preprocessed WET archives for content selection, and WARC for extraction. At the center stage sits our model based Markdown extractor, that is the primary workhorse to produce our dataset.
• Figure 1: Stats of experimental datasets SWEb and FineWeb
• Figure 2: Illustration of our proposed line classification model. Each newline is replaced by a special <s> token, and the corresponding embeddings are used for classification
• Figure 3: Precision/recall of our final line extraction model. We pick a threshold of 0.05 at inference, e.g. when applying the model for extraction.
• Figure 4: Filtering distributions on two Common Crawl dumps, and exclude regions marked in red. We exclude documents whose content length is shorter than 100 chars (invisible in the chart).