A Split-then-Join Approach to Abstractive Summarization for Very Long Documents in a Low Resource Setting

TL;DR

This work tackles long-document abstractive summarization under a fixed token budget by introducing SPIN, a split-then-join framework that enables a BIGBIRD-PEGASUS backbone to process documents far beyond $4,096$ tokens. It presents SPIN 1, SPIN 2, and SPIN 3 variants that partition inputs (and, in some cases, targets) and align parts using an $R_{LCS}$-based strategy, followed by a join step to reconstruct a full summary. Evaluations on long arXiv and BigPatent data show SPIN 3 delivers clear gains over the baseline and other SPIN variants, demonstrating that important content can reside throughout the document rather than at the beginning. The approach provides a practical, data-efficient path to high-quality long-document summaries in low-resource settings by leveraging partitioning and attention-efficient architectures.

Abstract

$\texttt{BIGBIRD-PEGASUS}$ model achieves $\textit{state-of-the-art}$ on abstractive text summarization for long documents. However it's capacity still limited to maximum of $4,096$ tokens, thus caused performance degradation on summarization for very long documents. Common method to deal with the issue is to truncate the documents. In this reasearch, we'll use different approach. We'll use the pretrained $\texttt{BIGBIRD-PEGASUS}$ model by fine tuned the model on other domain dataset. First, we filter out all documents which length less than $20,000$ tokens to focus on very long documents. To prevent domain shifting problem and overfitting on transfer learning due to small dataset, we augment the dataset by splitting document-summary training pair into parts, to fit the document into $4,096$ tokens. Source code available on $\href{https://github.com/lhfazry/SPIN-summ}{https://github.com/lhfazry/SPIN-summ}$.

Figures (1)

• Figure 1: Document-summary pairing strategy between SPIN 1 and SPIN 2 & 3