What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization

TL;DR

The paper introduces Knowledge-Grounded Discussion Summarization (KGDS), a task that pairs a background summary with an opinion summary to address reader confusion caused by omitted context in traditional dialogue summaries. It formalizes two paradigms (EBS-AOS and ABS-AOS), builds the first KGDS benchmark from 100 news discussions with multi-granularity gold annotations, and presents a hierarchical evaluation framework to assess background and opinion sub-summaries. Through extensive testing of 12 advanced LLMs under structured prompts and self-reflection, the study finds KGDS remains challenging: background retrieval and generation are imprecise, opinion integration struggles with implicit references, and self-reflection provides limited benefits. The work highlights concrete directions in retrieval, fine-grained generation, and knowledge integration to advance reader-centered summaries in knowledge-grounded discussions.

Abstract

Traditional dialogue summarization primarily focuses on dialogue content, assuming it comprises adequate information for a clear summary. However, this assumption often fails for discussions grounded in shared background, where participants frequently omit context and use implicit references. This results in summaries that are confusing to readers unfamiliar with the background. To address this, we introduce Knowledge-Grounded Discussion Summarization (KGDS), a novel task that produces a supplementary background summary for context and a clear opinion summary with clarified references. To facilitate research, we construct the first KGDS benchmark, featuring news-discussion pairs and expert-created multi-granularity gold annotations for evaluating sub-summaries. We also propose a novel hierarchical evaluation framework with fine-grained and interpretable metrics. Our extensive evaluation of 12 advanced large language models (LLMs) reveals that KGDS remains a significant challenge. The models frequently miss key facts and retain irrelevant ones in background summarization, and often fail to resolve implicit references in opinion summary integration.

Figures (18)

• Figure 1: An overview example. Gray Blocks in shared background knowledge denote crucial background details omitted by participants during discussion. Purple Blocks in discussion content indicate referential pronouns or phrases. Blue Blocks in discussion summary represent content that may cause confusion for outside readers. Compared to traditional dialogue summarization, KGDS achieves better reader preference by providing a supplementary background summary and a clear opinion summary, in which Cyan Blocks highlight clarified implicit references.
• Figure 2: An overview example of our evaluation framework. It comprehensively and accurately evaluates sub-summaries and summarization paradigms through fine-grained, interpretable metrics and hierarchical aggregation.
• Figure 3: Paragraph retrieval ratios ($\%$) of LLMs. The majority of models can be classified as either conservative ($ratio<30\hbox{$\%$}$) or open retrievers ($ratio>38\hbox{$\%$}$).
• Figure 4: Visualization and metrics of the correlation between $\mathrm{SP}_{\mathrm{F_1}}$ and $\mathrm{CAO}_{\mathrm{R}}$ under the EBS-AOS paradigm.
• Figure 5: Vis. and metrics of the correlation between $\mathrm{KSAF}_{\mathrm{F_1}}$ and $\mathrm{CAO}_{\mathrm{R}}$ under the ABS-AOS paradigm.