Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Second Look on BASS -- Boosting Abstractive Summarization with Unified Semantic Graphs -- A Replication Study

Osman Alperen Koraş, Jörg Schlötterer, Christin Seifert

TL;DR

This replication study investigates BASS, a graph-enhanced abstractive summarization framework based on Unified Semantic Graphs, by reimplementing its components and evaluating two graph-construction variants. The authors report substantial gaps between the original results and their replications, attributing much of the discrepancy to architectural details, undertraining, and inconsistencies between the paper and the provided code. An ablation program shows limited gains from the proposed graph adaptations, suggesting the original architecture may not be as effective as claimed or that the replication conditions differed significantly. The work underscores the importance of thorough documentation, version-controlled code sharing, and explicit reproducibility considerations to enable reliable comparisons in NLP research.

Abstract

We present a detailed replication study of the BASS framework, an abstractive summarization system based on the notion of Unified Semantic Graphs. Our investigation includes challenges in replicating key components and an ablation study to systematically isolate error sources rooted in replicating novel components. Our findings reveal discrepancies in performance compared to the original work. We highlight the significance of paying careful attention even to reasonably omitted details for replicating advanced frameworks like BASS, and emphasize key practices for writing replicable papers.

A Second Look on BASS -- Boosting Abstractive Summarization with Unified Semantic Graphs -- A Replication Study

TL;DR

This replication study investigates BASS, a graph-enhanced abstractive summarization framework based on Unified Semantic Graphs, by reimplementing its components and evaluating two graph-construction variants. The authors report substantial gaps between the original results and their replications, attributing much of the discrepancy to architectural details, undertraining, and inconsistencies between the paper and the provided code. An ablation program shows limited gains from the proposed graph adaptations, suggesting the original architecture may not be as effective as claimed or that the replication conditions differed significantly. The work underscores the importance of thorough documentation, version-controlled code sharing, and explicit reproducibility considerations to enable reliable comparisons in NLP research.

Abstract

We present a detailed replication study of the BASS framework, an abstractive summarization system based on the notion of Unified Semantic Graphs. Our investigation includes challenges in replicating key components and an ablation study to systematically isolate error sources rooted in replicating novel components. Our findings reveal discrepancies in performance compared to the original work. We highlight the significance of paying careful attention even to reasonably omitted details for replicating advanced frameworks like BASS, and emphasize key practices for writing replicable papers.
Paper Structure (35 sections, 2 equations, 1 figure, 3 tables)

This paper contains 35 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Replication Methodology
  3. Replicating the BASS framework
  4. Pre-processing ①.
  5. Graph Construction ②.
  6. Graph Augmentation ③.
  7. Text Encoder and Tokenizer ④.
  8. Aligning Text to Graph Embeddings ⑤.
  9. Model Architecture ⑥ -- ⑨.
  10. Evaluation
  11. Datasets.
  12. Models.
  13. Training Details.
  14. Evaluation.
  15. Experimental Results
  16. ...and 20 more sections

Figures (1)

  • Figure 1: Illustration of the BASS framework. The pre-processing and graph construction is done on the input document (left). The resulting graph information is used for token-to-node alignment ⑤, the graph encoder ⑥ and the respective cross-attention module ⑦ in the decoder.