HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering

TL;DR

The paper tackles the challenge of long-document summarization by addressing information dispersion and misleading narrative order in LLMs. It introduces HERA, a framework that first packs context by extracting event-relevant segments and then reorders them with a non-autoregressive model before generating and aggregating summaries. Experiments on arXiv and PubMed across multiple LLMs show notable gains in ROUGE, BERTScore, FactCC, and SummaC without additional training, indicating improved fluency and faithfulness. This approach offers a practical, model-agnostic enhancement for long-document summaries and provides ablation and hyperparameter analyses to guide future work.

Abstract

Despite the rapid growth of context length of large language models (LLMs) , LLMs still perform poorly in long document summarization. An important reason for this is that relevant information about an event is scattered throughout long documents, and the messy narrative order impairs the accurate understanding and utilization of LLMs for long documents. To address these issues, we propose a novel summary generation framework, called HERA. Specifically, we first segment a long document by its semantic structure and retrieve text segments about the same event, and finally reorder them to form the input context. We evaluate our approach on two long document summarization datasets. The experimental results show that HERA outperforms foundation models in ROUGE, BERTScore and faithfulness metrics, while HERA does not require additional fine-tuning and resources.

Figures (1)

• Figure 1: The pipeline of our proposed HERA approach.