Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries

TL;DR

The paper tackles the MDDS challenge where LLMs exhibit 'lost in the middle' attention and uneven source coverage. It proposes a principled three-step pipeline: extract atomic key points from each document, select a diverse and (when needed) user-relevant subset via Determinantal Point Processes (DPPs), and rewrite the chosen points into a final summary, avoiding purely one-step prompting. Empirical results on DiverseSumm (and augmented variants) show that LLM + DPP yields superior source coverage across multiple models, and relevance-aware DPPs improve query-focused coverage. This approach demonstrates a practical path to robust, personalized multi-document summaries in agentic workflows, without requiring heavy model fine-tuning, by leveraging principled content selection in combination with strong generation capabilities.

Abstract

While large language models (LLMs) are increasingly capable of handling longer contexts, recent work has demonstrated that they exhibit the "lost in the middle" phenomenon (Liu et al., 2024) of unevenly attending to different parts of the provided context. This hinders their ability to cover diverse source material in multi-document summarization, as noted in the DiverseSumm benchmark (Huang et al., 2024). In this work, we contend that principled content selection is a simple way to increase source coverage on this task. As opposed to prompting an LLM to perform the summarization in a single step, we explicitly divide the task into three steps -- (1) reducing document collections to atomic key points, (2) using determinantal point processes (DPP) to perform select key points that prioritize diverse content, and (3) rewriting to the final summary. By combining prompting steps, for extraction and rewriting, with principled techniques, for content selection, we consistently improve source coverage on the DiverseSumm benchmark across various LLMs. Finally, we also show that by incorporating relevance to a provided user intent into the DPP kernel, we can generate personalized summaries that cover relevant source information while retaining coverage.

Figures (6)

• Figure 1: Overview of the MDDS task (\ref{['sec:formulation']}), which aims to generate a summary from a set of source documents with an optional user intent. Compared to (a) prompting an LLM to perform MDDS in a single step (Naive LLM) and other baselines, (b) our method (LLM + DPP) first extracts atomic key points from each document, then explicitly selects content using DPPs to ensure diversity and relevance before rewriting them into a summary (\ref{['sec:method']}). LLM + DPP improves source coverage and produces summaries more aligned with user intent (\ref{['sec:results']}).
• Figure 2: Studying the 'lost-in-the-middle' phenomenon by plotting coverage of different source articles by index with Naive LLM and LLM + DPP. While Naive LLM exhibits biases to better cover the articles at the start (GPT-4o, GPT-3.5) or end (Llama) of the context, LLM + DPP has higher and more uniform coverage of all source documents---mitigating these biases.
• Figure 3: Distribution of source documents covered by key points when selected with LLM + DPP and LLM-Selected KPs. LLM + DPP exhibits consistently higher coverage of source documents.
• Figure 4: Case study of LLM + DPP (\ref{['sec:relevance-relevance']}) selecting key points that are diverse and yet relevant to two different user intents (\ref{['sec:kp_rel_selection']}) and evaluation of the summaries via question-answering (\ref{['sec:evaluation_method']}).
• Figure 5: TSNE visualization of the key points selected for the two user intents in \ref{['fig:example_relevance']} from the document set. Blue triangles represent selected key points, while red circles denote unselected points. Color intensity reflects relevance to the respective user intent. LLM + DPP is able to select relevant key points while also prioritizing diverse coverage of the source material.