Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Rank Salient Content for Query-focused Summarization

Sajad Sotudeh, Nazli Goharian

TL;DR

A deeper analysis reveals the model’s superiority over the state-of-the-art for broad queries, as opposed to specific ones, from a qualitative standpoint, and presents an error analysis of the model.

Abstract

This study examines the potential of integrating Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization. Using a shared secondary decoder with the summarization decoder, we carry out the LTR task at the segment level. Compared to the state-of-the-art, our model outperforms on QMSum benchmark (all metrics) and matches on SQuALITY benchmark (2 metrics) as measured by Rouge and BertScore while offering a lower training overhead. Specifically, on the QMSum benchmark, our proposed system achieves improvements, particularly in Rouge-L (+0.42) and BertScore (+0.34), indicating enhanced understanding and relevance. While facing minor challenges in Rouge-1 and Rouge-2 scores on the SQuALITY benchmark, the model significantly excels in Rouge-L (+1.47), underscoring its capability to generate coherent summaries. Human evaluations emphasize the efficacy of our method in terms of relevance and faithfulness of the generated summaries, without sacrificing fluency. A deeper analysis reveals our model's superiority over the state-of-the-art for broad queries, as opposed to specific ones, from a qualitative standpoint. We further present an error analysis of our model, pinpointing challenges faced and suggesting potential directions for future research in this field.

Learning to Rank Salient Content for Query-focused Summarization

TL;DR

A deeper analysis reveals the model’s superiority over the state-of-the-art for broad queries, as opposed to specific ones, from a qualitative standpoint, and presents an error analysis of the model.

Abstract

This study examines the potential of integrating Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization. Using a shared secondary decoder with the summarization decoder, we carry out the LTR task at the segment level. Compared to the state-of-the-art, our model outperforms on QMSum benchmark (all metrics) and matches on SQuALITY benchmark (2 metrics) as measured by Rouge and BertScore while offering a lower training overhead. Specifically, on the QMSum benchmark, our proposed system achieves improvements, particularly in Rouge-L (+0.42) and BertScore (+0.34), indicating enhanced understanding and relevance. While facing minor challenges in Rouge-1 and Rouge-2 scores on the SQuALITY benchmark, the model significantly excels in Rouge-L (+1.47), underscoring its capability to generate coherent summaries. Human evaluations emphasize the efficacy of our method in terms of relevance and faithfulness of the generated summaries, without sacrificing fluency. A deeper analysis reveals our model's superiority over the state-of-the-art for broad queries, as opposed to specific ones, from a qualitative standpoint. We further present an error analysis of our model, pinpointing challenges faced and suggesting potential directions for future research in this field.

Paper Structure

This paper contains 21 sections, 6 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Background: Segment Summarizer (SegEnc)
  4. Model: LTR-assisted Summarization
  5. Experimental Setup
  6. Research questions
  7. Datasets
  8. Relevance labeling
  9. Implementation details
  10. Comparison
  11. Experimental Results
  12. Automatic evaluation
  13. Ablation study
  14. Human evaluation
  15. Query type impact
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Overview of our proposed system (i.e., LTRSum). Our system utilizes a shared-parameter decoder across two tasks, for the sake of learning to rank source segments (right-side decoder) alongside summarization (left-side decoder). It is important to note that our system uses a single decoder that shares parameters across both tasks, but for visual clarity, two decoders are depicted here.
  • Figure 2: Segment retrieval performance of the models in terms of nDCG score.