EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents

TL;DR

The paper tackles the challenge of socio-temporal event understanding by proposing EventSum, the first large-scale Chinese event-centric multi-document summarization dataset built from Baidu Baike entries. It combines automatic data construction with manual test-set annotation to enable robust evaluation of dynamic event summaries across multiple related news documents. To address the inadequacy of traditional metrics, the authors design recall-based metrics for Event, Argument, Causal, and Temporal information, evaluated via NLI discriminators. Experimental results using long-context LLMs reveal that ECS remains difficult and that the designed recall metrics provide meaningful insight into summary completeness. EventSum thus offers a valuable resource for advancing event-centric MDS and long-context reasoning in Chinese, with potential extension to English and longer time-span events in the future.

Abstract

In real life, many dynamic events, such as major disasters and large-scale sports events, evolve continuously over time. Obtaining an overview of these events can help people quickly understand the situation and respond more effectively. This is challenging because the key information of the event is often scattered across multiple documents, involving complex event knowledge understanding and reasoning, which is under-explored in previous work. Therefore, we proposed the Event-Centric Multi-Document Summarization (ECS) task, which aims to generate concise and comprehensive summaries of a given event based on multiple related news documents. Based on this, we constructed the EventSum dataset, which was constructed using Baidu Baike entries and underwent extensive human annotation, to facilitate relevant research. It is the first large scale Chinese multi-document summarization dataset, containing 5,100 events and a total of 57,984 news documents, with an average of 11.4 input news documents and 13,471 characters per event. To ensure data quality and mitigate potential data leakage, we adopted a multi-stage annotation approach for manually labeling the test set. Given the complexity of event-related information, existing metrics struggle to comprehensively assess the quality of generated summaries. We designed specific metrics including Event Recall, Argument Recall, Causal Recall, and Temporal Recall along with corresponding calculation methods for evaluation. We conducted comprehensive experiments on EventSum to evaluate the performance of advanced long-context Large Language Models (LLMs) on this task. Our experimental results indicate that: 1) The event-centric multi-document summarization task remains challenging for existing long-context LLMs; 2) The recall metrics we designed are crucial for evaluating the comprehensiveness of the summary information.

Figures (9)

• Figure 1: An example of the "2023 Hebei Heavy Rain". The right side of the figure shows the English translation obtained from the original documents on the left.
• Figure 2: Overview of the construction process. It introduces the data construction process for the "2023 Hebei Heavy Rain" event from the retrieved entries for "Events of July 2023" on the Baidu Baike website.
• Figure 3: Analysis of Input Documents. The distribution of the number of input documents (a) and the total input characters (b) are presented on the left and right, respectively.
• Figure 4: Analysis of Reference Summary. The distribution of characters in the reference summary is shown on the left (a), while the distribution of the dynamic event time span within the reference summary is displayed on the right (b).
• Figure 5: Analysis of the impact of various input documents number (left) and time span (right) of the dynamic event.