Write Summary Step-by-Step: A Pilot Study of Stepwise Summarization

TL;DR

The paper tackles stepwise summarization, enabling appended summaries as new documents arrive while preserving coherence with prior summaries. It introduces the Stepwise Summary Generator (SSG), consisting of a stream-aware encoder, a Transformer-based summary generator, and a coherence discriminator to enforce consistency across steps. A large-scale Stepwise Summarization Dataset (SSD) is released to benchmark this setting, and extensive experiments show SSG achieves state-of-the-art performance on automatic metrics and human judgments, with ablations validating the contributions of each module. The work highlights the practical impact for evolving information streams and discusses the role of large language models in future improvements.

Abstract

Nowadays, neural text generation has made tremendous progress in abstractive summarization tasks. However, most of the existing summarization models take in the whole document all at once, which sometimes cannot meet the needs in practice. Practically, social text streams such as news events and tweets keep growing from time to time, and can only be fed to the summarization system step by step. Hence, in this paper, we propose the task of Stepwise Summarization, which aims to generate a new appended summary each time a new document is proposed. The appended summary should not only summarize the newly added content but also be coherent with the previous summary, to form an up-to-date complete summary. To tackle this challenge, we design an adversarial learning model, named Stepwise Summary Generator (SSG). First, SSG selectively processes the new document under the guidance of the previous summary, obtaining polished document representation. Next, SSG generates the summary considering both the previous summary and the document. Finally, a convolutional-based discriminator is employed to determine whether the newly generated summary is coherent with the previous summary. For the experiment, we extend the traditional two-step update summarization setting to a multi-step stepwise setting, and re-propose a large-scale stepwise summarization dataset based on a public story generation dataset. Extensive experiments on this dataset show that SSG achieves state-of-the-art performance in terms of both automatic metrics and human evaluations. Ablation studies demonstrate the effectiveness of each module in our framework. We also discuss the benefits and limitations of recent large language models on this task.

Figures (4)

• Figure 1: Comparison between stream summarization, update summarization, and stepwise summarization. Stream summarization Ge2016NewsSS: a large number documents are taken as input, and some of the documents are selected as summary. Update summarization Dang2008OverviewOT: the task is to summarize the new document, assuming the reader already knows about previous documents. Stepwise summarization: This setting does not require the reader to have any prior knowledge about the text stream. The generated summary contains all the information about the text stream, and is kept fluent all the time steps.
• Figure 2: Illustration of a sample in the stepwise summarization dataset.
• Figure 3: Overview of SSG. We divide our model into three parts: (1) Stream-aware Encoder (left) selectively processes document under guidance from the previous summary. (2) Summary generator (middle) generates a summary taking both factors above into consideration. (3) Coherence discriminator (right) determines whether the generated summary is coherent with the previous one.
• Figure 4: (a) Influence of previous information (DP score) on summarization performance (DS score). (b) Error accumulation analysis when the text stream gets longer.