Faithful Summarisation under Disagreement via Belief-Level Aggregation
Favour Yahdii Aghaebe, Tanefa Apekey, Elizabeth Williams, Nafise Sadat Moosavi
TL;DR
The paper tackles faithful summarisation in the presence of genuine disagreement by isolating aggregation from generation. It introduces AccSynth, a three-stage pipeline that represents each document as a structured belief base $K_i$, merges these using a distance-based operator $\Delta^{d,f}$ to produce a merged base $K^{\ast}$, and realises the final summary from $K^{\ast}$ with lightweight prompts. Across model families and scales (Qwen, LLaMA, GPT), belief-level aggregation provides stable, disagreement-aware performance with simple prompting, while generation-time fusion is highly model-dependent. The results suggest that explicit, principled aggregation can substitute for larger models and complex synthesis prompts, yielding more faithful summaries in opinion-rich domains such as movie reviews.
Abstract
Opinion and multi-document summarisation often involve genuinely conflicting viewpoints, yet many existing approaches, particularly LLM-based systems, implicitly smooth disagreement and over-represent majority opinions. This limits the faithfulness of generated summaries in opinion-heavy settings. We introduce a disagreement-aware synthesis pipeline that separates belief-level aggregation from language generation. Documents are first represented as structured belief sets and aggregated using distance-based belief merging operators that explicitly model conflict. Large language models are then used only to realise the aggregated beliefs as natural language summaries. We evaluate the approach across multiple model families and scales, comparing it to methods that perform explicit aggregation during generation. Our results show that while sufficiently large models can match belief-level aggregation when aggregation is handled at generation time, this behaviour is not stable across architectures or capacities. In contrast, belief-level aggregation combined with simple prompting yields consistently strong disagreement-aware performance across models, while maintaining fluent and grounded summaries.