CADS: A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization

TL;DR

This systematic review introduces the CADS taxonomy to unify the six core challenges in abstractive dialogue summarization and maps Transformer-based approaches to these challenges across English dialogue data from 2019–2024. It synthesizes 133 carefully selected papers, detailing techniques such as graph-based models, AMR representations, and multi-stage segmentation to address language, structure, comprehension, speaker, salience, and factuality. The review catalogues 18 datasets by subdomain, analyzes data augmentation strategies to mitigate data scarcity, and surveys evaluation practices, highlighting heavy reliance on ROUGE and underreporting of human evaluation details. Finally, it discusses the implications of large language models and confirms the ongoing relevance of the CADS taxonomy, offering guidance for future research on datasets, evaluation, and cross-domain generalization.

Abstract

Abstractive dialogue summarization is the task of distilling conversations into informative and concise summaries. Although reviews have been conducted on this topic, there is a lack of comprehensive work detailing the challenges of dialogue summarization, unifying the differing understanding of the task, and aligning proposed techniques, datasets, and evaluation metrics with the challenges. This article summarizes the research on Transformer-based abstractive summarization for English dialogues by systematically reviewing 1262 unique research papers published between 2019 and 2024, relying on the Semantic Scholar and DBLP databases. We cover the main challenges present in dialog summarization (i.e., language, structure, comprehension, speaker, salience, and factuality) and link them to corresponding techniques such as graph-based approaches, additional training tasks, and planning strategies, which typically overly rely on BART-based encoder-decoder models. We find that while some challenges, like language, have seen considerable progress, mainly due to training methods, others, such as comprehension, factuality, and salience, remain difficult and hold significant research opportunities. We investigate how these approaches are typically assessed, covering the datasets for the subdomains of dialogue (e.g., meeting, medical), the established automatic metrics and human evaluation approaches for assessing scores and annotator agreement. We observe that only a few datasets span across all subdomains. The ROUGE metric is the most used, while human evaluation is frequently reported without sufficient detail on inner-annotator agreement and annotation guidelines. Additionally, we discuss the possible implications of the recently explored large language models and conclude that despite a potential shift in relevance and difficulty, our described challenge taxonomy remains relevant.

Figures (8)

• Figure 1: Overview of the six challenges in dialogue summarization, including a brief description of each challenge and an estimation of progress for related sub-challenges. Progress is evaluated based on two factors: (1) the extent to which mitigation strategies have been developed, and (2) the measurable improvement in summary quality as a result of these strategies. Green means mostly mitigated, orange means good progress, and red stands for marked challenges still exist.
• Figure 2: An overview of the different stages of our methodology and their inherent sub-stages. The red boxes indicate the number of papers considered in the respective step.
• Figure 3: Dialogue snippet showing examples of the idiosyncratic nature of spoken language and individual speech patterns. Red displays disfluencies, blue highlights personal speech patterns, orange stands for colloquialism, green represent informal expressions.
• Figure 4: Excerpt from a conversation to demonstrate the structural challenge, with a detailed view on one topic out of multiple topics touched in the whole conversation. The red arcs mark the connection between individual turns, demonstrating how different dialogue phases (grey text), such as the conclusion, relate to many previous conversation phases.
• Figure 5: Example conversation to demonstrate the comprehension challenge including direct and implied content. Directly stated (orange) are the explicit updates by Bob and Charlie about their respective tasks. Green marks implied content such as a deadline for a demo and a current focus on performance and finalizing features (unmentioned background knowledge), Redis implies an understanding of its role in performance enhancement (organizational knowledge and reference to prior discussions on data requirements).