Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation

TL;DR

The paper tackles cross-lingual cross-temporal summarization (CLCTS) by constructing the first CLCTS corpus linking historical German-English and English-German texts with modern Wikipedia summaries, and by exploring pipeline extractive, end-to-end transformer, and GPT-based approaches. It evaluates these methods using automatic metrics, human judgments, and GPT-4 as an evaluator, finding that intermediate-task finetuned end-to-end models underperform, while zero-shot GPT-3.5 summaries are often moderate to good and capable of historical normalization; GPT-4 aligns with human judgments to a moderate degree. The study further analyzes data characteristics—longer, older texts with greater linguistic divergence are harder to summarize—and reveals limited gains from modest dataset expansion. It also investigates data contamination concerns and the potential and limitations of ChatGPT in CLCTS, proposing future directions that include larger, more diverse data, multi-task pretraining, and advanced long-document architectures to advance cross-cultural access to historical information.

Abstract

While summarization has been extensively researched in natural language processing (NLP), cross-lingual cross-temporal summarization (CLCTS) is a largely unexplored area that has the potential to improve cross-cultural accessibility and understanding. This paper comprehensively addresses the CLCTS task, including dataset creation, modeling, and evaluation. We (1) build the first CLCTS corpus with 328 instances for hDe-En (extended version with 455 instances) and 289 for hEn-De (extended version with 501 instances), leveraging historical fiction texts and Wikipedia summaries in English and German; (2) examine the effectiveness of popular transformer end-to-end models with different intermediate finetuning tasks; (3) explore the potential of GPT-3.5 as a summarizer; (4) report evaluations from humans, GPT-4, and several recent automatic evaluation metrics. Our results indicate that intermediate task finetuned end-to-end models generate bad to moderate quality summaries while GPT-3.5, as a zero-shot summarizer, provides moderate to good quality outputs. GPT-3.5 also seems very adept at normalizing historical text. To assess data contamination in GPT-3.5, we design an adversarial attack scheme in which we find that GPT-3.5 performs slightly worse for unseen source documents compared to seen documents. Moreover, it sometimes hallucinates when the source sentences are inverted against its prior knowledge with a summarization accuracy of 0.67 for plot omission, 0.71 for entity swap, and 0.53 for plot negation. Overall, our regression results of model performances suggest that longer, older, and more complex source texts (all of which are more characteristic for historical language variants) are harder to summarize for all models, indicating the difficulty of the CLCTS task.

Figures (6)

• Figure 1: Texts matching from DTA, Wikisource, and Project Gutenberg to summaries collected from Wikipedia. Non-paired texts are excluded (highlighted in red).
• Figure 2: Distribution of publication year.
• Figure 3: Mean dependency distance (MDD) for document sentences in each dataset. Historical documents are marked with red stars.
• Figure 4: Flow chart illustrating models and evaluation strategies in Section \ref{['sec:setup']}.
• Figure 5: Segment level Spearman's ranking correlation between human annotation and evaluation metrics. The corresponding p-value is obtained through a two-tailed Student's t-test at significance levels 0.05 (*), 0.01 (**), and 0.001(***).