Hierarchical Attention Graph for Scientific Document Summarization in Global and Local Level

TL;DR

This work tackles extractive summarization of long scientific documents by modeling both local intra-sentence relations and global high-order inter-sentence relations within a hierarchical discourse framework. It presents HAESum, a two-stage architecture combining a local heterogeneous graph (HEGAT) for sentence-word interactions and a global hypergraph self-attention network (HGSAT) for cross-sentence relations, without relying on pre-trained encoders. The approach achieves state-of-the-art ROUGE scores on PubMed and Arxiv, and ablation studies confirm the necessity of both components and hierarchical updating. The findings underscore the importance of hierarchical structure in long-document summarization and hint at practical benefits for low-resource languages and future knowledge-augmented, multi-document extensions.

Abstract

Scientific document summarization has been a challenging task due to the long structure of the input text. The long input hinders the simultaneous effective modeling of both global high-order relations between sentences and local intra-sentence relations which is the most critical step in extractive summarization. However, existing methods mostly focus on one type of relation, neglecting the simultaneous effective modeling of both relations, which can lead to insufficient learning of semantic representations. In this paper, we propose HAESum, a novel approach utilizing graph neural networks to locally and globally model documents based on their hierarchical discourse structure. First, intra-sentence relations are learned using a local heterogeneous graph. Subsequently, a novel hypergraph self-attention layer is introduced to further enhance the characterization of high-order inter-sentence relations. We validate our approach on two benchmark datasets, and the experimental results demonstrate the effectiveness of HAESum and the importance of considering hierarchical structures in modeling long scientific documents. Our code will be available at \url{https://github.com/MoLICHENXI/HAESum}

Figures (3)

• Figure 1: An illustration of modeling an input document from local and global perspectives. Triangles and circles represent words and sentences in the original document respectively.
• Figure 2: Overview of the proposed HAESum framework. We first build a local-level heterogeneous graph (HEGAT) for the input document and apply message passing to iteratively update the representation in two stages: sentence-to-word and word-to-sentence. The obtained sentence representations are then fed into the hypergraph self-attention layer (HGSAT) to obtain the global representations and used for the final sentence selection.
• Figure 3: ROUGE-1,2 performance of HAESum with different number of graph nodes on PubMed dataset.