Cross-Document Event-Keyed Summarization

TL;DR

This work extends event-keyed summarization (EKS) to cross-document settings by introducing SEAMuS, an expert-annotated dataset derived from FAMuS CDAE annotations to support single- and cross-document event-centered summaries. It benchmarks a range of models—from fine-tuned encoder-decoder architectures to zero-/few-shot prompting LLMs—across report-only and cross-document tasks, supplemented by extensive input ablations and robustness analyses under extraction noise. The results show that while large models and few-shot prompting improve performance over a report baseline, cross-document summarization remains more challenging, with robust performance achievable through careful input structure (Text+Event) and retrieval-based context selection. Human evaluation confirms generally high-quality outputs across models but reveals variability across raters, underscoring the need for task-oriented evaluation and user-specific preferences. SEAMuS is released to spur further research into reliable, event-focused synthesis across documents, with planned expansions to more sources and larger-scale annotation efforts.

Abstract

Event-keyed summarization (EKS) requires summarizing a specific event described in a document given the document text and an event representation extracted from it. In this work, we extend EKS to the cross-document setting (CDEKS), in which summaries must synthesize information from accounts of the same event as given by multiple sources. We introduce SEAMUS (Summaries of Events Across Multiple Sources), a high-quality dataset for CDEKS based on an expert reannotation of the FAMUS dataset for cross-document argument extraction. We present a suite of baselines on SEAMUS -- covering both smaller, fine-tuned models, as well as zero- and few-shot prompted LLMs -- along with detailed ablations and a human evaluation study, showing SEAMUS to be a valuable benchmark for this new task.

Figures (5)

• Figure 1: Schematic illustration of the SEAMuS report and cross-document event-keyed summarization tasks. Letters represent event arguments.
• Figure 2: An example from our SEAMuS dataset. Report documents (bottom left) are Wikipedia passages that describe some event (top right) and that cite a longer (non-Wikipedia) source article (bottom right) as evidence, with event arguments annotated in both documents. SEAMuS features simple summaries of these events based on only the report (top left) as well as enriched, cross-document summaries based on both the report and its source, which typically contain additional information about the event (here, the Crime). \ref{['app:examples']} has further examples.
• Figure 3: Histograms of summary quality scores (1-5, higher is better) from our human evaluation (§\ref{['sec:analysis']}). The bottom right plot (red) aggregates scores across all three raters; each of the other plots (blue) shows a single rater's scores.
• Figure 4: Interface for source text argument correction and cross-document summary writing (the first part of the Phase 2 annotation).
• Figure 5: Interface for annotation of arguments on the cross-document summaries (the second part of the Phase 2 annotation).