CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs

TL;DR

CrossSum tackles cross-lingual summarization beyond English by constructing a large-scale, non-English XLS dataset across 1,500+ language pairs via cross-lingual retrieval on XL-Sum. It introduces induced pairs and leakage mitigation to improve data quality, a multistage language sampling (MLS) strategy for training a many-to-many model, and LaSE, a language-agnostic evaluation metric. Empirical results show MLS-enabled many-to-many models outperform traditional baselines, with LaSE showing strong correlation to ROUGE and robustness when target-language references are unavailable. The work releases dataset, code, and models to foster inclusive XLS research and discusses zero-shot/few-shot directions and limitations.

Abstract

We present CrossSum, a large-scale cross-lingual summarization dataset comprising 1.68 million article-summary samples in 1,500+ language pairs. We create CrossSum by aligning parallel articles written in different languages via cross-lingual retrieval from a multilingual abstractive summarization dataset and perform a controlled human evaluation to validate its quality. We propose a multistage data sampling algorithm to effectively train a cross-lingual summarization model capable of summarizing an article in any target language. We also introduce LaSE, an embedding-based metric for automatically evaluating model-generated summaries. LaSE is strongly correlated with ROUGE and, unlike ROUGE, can be reliably measured even in the absence of references in the target language. Performance on ROUGE and LaSE indicate that our proposed model consistently outperforms baseline models. To the best of our knowledge, CrossSum is the largest cross-lingual summarization dataset and the first ever that is not centered around English. We are releasing the dataset, training and evaluation scripts, and models to spur future research on cross-lingual summarization. The resources can be found at https://github.com/csebuetnlp/CrossSum

Figures (12)

• Figure 1: A sample article-summary pair from CrossSum, the article is written in Japanese, and the summary is in Bengali. We translate the texts to English inside parentheses for better understanding. Words and phrases of the article relevant to the summary are color-coded.
• Figure 2: Training on the dataset respecting the original XL-Sum splits causes unusually high ROUGE scores (marked red) in many-to-one models due to implicit data leakage. Therefore, we redid the splits taking the issue into account, and consequently, models trained on the new set (marked blue) do not exhibit any unusual spike.
• Figure 3: A heatmap showing alignment accuracies of different language pairs obtained by human evaluation.
• Figure 4: ROUGE-2 and LaSE scores for English and Chinese as target languages as the source languages vary. The m2m model significantly outperforms the m2o models and summarize-then-translate baseline in most languages. The comparisons with other target languages are shown in the Appendix (Figure \ref{['fig:m2o-app']}) due to space limitations.
• Figure 5: ROUGE-2 and LaSE scores for English and Chinese as source languages as the target languages vary. The m2m model significantly outperforms the o2m models and summarize-then-translate baseline in most languages. The comparisons with other source languages are shown in the Appendix (Figure \ref{['fig:o2m-app']}) due to space limitations.