Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts

Xiaobo Guo, Soroush Vosoughi

TL;DR

This work introduces Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text and reveals that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.

Abstract

Aspect-based summarization has seen significant advancements, especially in structured text. Yet, summarizing disordered, large-scale texts, like those found in social media and customer feedback, remains a significant challenge. Current research largely targets predefined aspects within structured texts, neglecting the complexities of dynamic and disordered environments. Addressing this gap, we introduce Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text. Developed by adapting existing datasets for cost-efficiency and scalability, our comprehensive experiments and detailed human evaluations reveal that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.

Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts

TL;DR

This work introduces Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text and reveals that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.

Abstract

Aspect-based summarization has seen significant advancements, especially in structured text. Yet, summarizing disordered, large-scale texts, like those found in social media and customer feedback, remains a significant challenge. Current research largely targets predefined aspects within structured texts, neglecting the complexities of dynamic and disordered environments. Addressing this gap, we introduce Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text. Developed by adapting existing datasets for cost-efficiency and scalability, our comprehensive experiments and detailed human evaluations reveal that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.
Paper Structure (35 sections, 1 equation, 8 figures, 10 tables)

This paper contains 35 sections, 1 equation, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dynamic Aspect-Based Summarization
  4. Automatic Evaluation Metrics
  5. Human Evaluation Metrics
  6. Datasets
  7. Dataset Creation
  8. Dataset Quality
  9. Baseline Models
  10. Clustering-Based-Summarization
  11. Keyword-Guided Summarization
  12. Prompt-Guided Summarization
  13. Evaluation and Results
  14. Automatic Evaluation Results
  15. Human Evaluations
  16. ...and 20 more sections

Figures (8)

  • Figure 1: An example from Reddit showcasing a multi-user discussion. Each participant's input is color-coded to distinguish the varied aspects they discuss, emphasizing the range of perspectives and topics within the conversation. This visual differentiation serves to highlight the diversity inherent in such online discussions.
  • Figure 2: Performance variation of baselines with changes in aspect number in the reference. The final data point represents cases where the number of aspects in the reference summary equals or exceeds 12 for D-CnnDM and 16 for D-WikiHow.
  • Figure A1: Distributions of article length, summary length, and aspect number across datasets. Mean and median values are marked with red and blue lines, respectively. For clarity, distributions are shown within three standard deviations from the mean for source articles and references.
  • Figure A2: Examples from the D-CnnDM, illustrating various aspects represented by different colors.
  • Figure A3: Examples from the D-WikiHow, illustrating various aspects represented by different colors.
  • ...and 3 more figures