A Novel LLM-based Two-stage Summarization Approach for Long Dialogues

TL;DR

Long-document summarization is hampered by model input limits and high compute costs. The paper proposes a two-stage hierarchical framework that first segments and condenses long texts using unsupervised topic segmentation and zero-shot prompting (producing first-stage summaries and event lists), then fine-tunes an abstractive model on the condensed data. Key contributions include a practical segmentation-condensation-training pipeline, analysis of ChatGPT-based condensation prompts, and validation on the ForeverDreaming dataset showing competitive ROUGE scores with reduced VRAM requirements. The approach offers a scalable solution for resource-constrained settings and can adapt across datasets with dataset-specific prompting strategies.

Abstract

Long document summarization poses a significant challenge in natural language processing due to input lengths that exceed the capacity of most state-of-the-art pre-trained language models. This study proposes a hierarchical framework that segments and condenses information from long documents, subsequently fine-tuning the processed text with an abstractive summarization model. Unsupervised topic segmentation methods identify semantically appropriate breakpoints. The condensation stage utilizes an unsupervised generation model to generate condensed data, and our current experiments employ ChatGPT(v3.5). The summarization stage fine-tunes the abstractive summarization model on the condensed data to generate the final results. This framework enables long documents to be processed on models even when the document length exceeds the model's maximum input size. The exclusion of the entire document from the summarization model reduces the time and computational resources required for training, making the framework suitable for contexts with constrained local computational resources.

Figures (1)

• Figure 1: Workflow of the proposed framework.