JADS: A Framework for Self-supervised Joint Aspect Discovery and Summarization

TL;DR

This paper tackles the challenge of generating multi-aspect summaries from non-continuous text with an unknown number of topics. It introduces Joint Aspect Discovery and Summarization (JADS), an end-to-end differentiable framework built on a Longformer encoder-decoder that simultaneously discovers topics and generates per-topic summaries; it is trained via self-supervised dataset construction using Wikipedia and CNN/DM data, with pretraining that improves performance and stability. The authors show that JADS outperforms traditional two-step baselines, yields embeddings with superior clustering capability, and aligns more closely with ground truth while remaining factual. The work advances open-world summarization by eliminating the need for predefined aspects and enabling scalable, topic-aware summaries applicable to diverse text collections, albeit with limitations related to topic-count bias and memory constraints for large K.

Abstract

To generate summaries that include multiple aspects or topics for text documents, most approaches use clustering or topic modeling to group relevant sentences and then generate a summary for each group. These approaches struggle to optimize the summarization and clustering algorithms jointly. On the other hand, aspect-based summarization requires known aspects. Our solution integrates topic discovery and summarization into a single step. Given text data, our Joint Aspect Discovery and Summarization algorithm (JADS) discovers aspects from the input and generates a summary of the topics, in one step. We propose a self-supervised framework that creates a labeled dataset by first mixing sentences from multiple documents (e.g., CNN/DailyMail articles) as the input and then uses the article summaries from the mixture as the labels. The JADS model outperforms the two-step baselines. With pretraining, the model achieves better performance and stability. Furthermore, embeddings derived from JADS exhibit superior clustering capabilities. Our proposed method achieves higher semantic alignment with ground truth and is factual.

Figures (4)

• Figure 1: JADS task compared to other summarization tasks.
• Figure 2: The performance of models trained and tested with various summary number
• Figure 3: Distribution of responses indicating which model's summary is more aligned with the ground truth.
• Figure 4: Assessment of factual accuracy in generated summaries across different options. 'Yes' indicates factual alignment, while 'No' indicates a deviation from factual content.