20min-XD: A Comparable Corpus of Swiss News Articles

TL;DR

20min-XD delivers a French-German document-level comparable corpus of Swiss news, comprising ~15k article pairs (2015–2024) and a sentence-aligned version with ~117k sentences, built from automatic semantic-alignment using title+lead signals. The authors compare multiple multilingual models and alignment strategies, selecting paraphrase-multilingual-mpnet with an intersection constraint and a threshold $ heta=46$ based on a small manually-curated validation set, and they supplement document-level analysis with AlignRatio, sentence-length, and monotonicity metrics. The dataset spans near-translation to loosely related content, enabling cross-lingual NLP tasks and linguistically motivated studies, and is released with code for reproducibility. The work also outlines future directions including full-text similarity, long-context multilingual embeddings, and cross-lingual difference recognition to extend the dataset’s utility.

Abstract

We present 20min-XD (20 Minuten cross-lingual document-level), a French-German, document-level comparable corpus of news articles, sourced from the Swiss online news outlet 20 Minuten/20 minutes. Our dataset comprises around 15,000 article pairs spanning 2015 to 2024, automatically aligned based on semantic similarity. We detail the data collection process and alignment methodology. Furthermore, we provide a qualitative and quantitative analysis of the corpus. The resulting dataset exhibits a broad spectrum of cross-lingual similarity, ranging from near-translations to loosely related articles, making it valuable for various NLP applications and broad linguistically motivated studies. We publicly release the dataset in document- and sentence-aligned versions and code for the described experiments.

Figures (5)

• Figure 1: Matrix visualization of different alignment strategies.
• Figure 2: Document (cosine) similarity score distribution over all 74,085 article pairs divided into 100 bins ranging from the threshold of 46 to 100. The dashed line indicates the cut above which the top 15,000 article pairs form the final comparable dataset.
• Figure 3: The document cosine similarities in comparison to the AlignRatio of each aligned article in German and French. Both languages show a positive trend line with weak positive correlation (FR: Pearson correlation coefficient $r = 0.145$; DE: $r = 0.103$).
• Figure 4: The document cosine similarities in comparison to the sentence length correlation of each aligned article. There is a very weak positive trend of correlation detectable between the two variables (Pearson correlation coefficient $r = 0.084$).
• Figure 5: The document cosine similarities in comparison to the monotonicity score of each aligned article. A weak positive correlation trend is detectable between the two variables (Pearson correlation coefficient $r = 0.147$).