From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization

TL;DR

This paper addresses the occurrence of hallucinations in large language models during multi-document summarization (MDS). It introduces two benchmarks derived from SummHay-News and SummHay-Conv to quantify topic-specific hallucinations as the number of input documents increases, and it evaluates five prominent LLMs using an LLM-as-judge evaluation framework. The results show substantial hallucination rates (up to 75% in conversations) and reveal that many errors stem from failing to follow instructions or producing overly generic insights, with non-existent-topic prompts further exposing fabrication tendencies. Simple post-hoc mitigation offers only marginal gains and often trades off recall, underscoring the need for more robust, systematic approaches to mitigate hallucinations in MDS. The work provides valuable datasets and analyses that will inform future mechanisms for grounding and evaluating multi-document summaries in practical applications.

Abstract

Although many studies have investigated and reduced hallucinations in large language models (LLMs) for single-document tasks, research on hallucination in multi-document summarization (MDS) tasks remains largely unexplored. Specifically, it is unclear how the challenges arising from handling multiple documents (e.g., repetition and diversity of information) affect models outputs. In this work, we investigate how hallucinations manifest in LLMs when summarizing topic-specific information from multiple documents. Since no benchmarks exist for investigating hallucinations in MDS, we use existing news and conversation datasets, annotated with topic-specific insights, to create two novel multi-document benchmarks. When evaluating 5 LLMs on our benchmarks, we observe that on average, up to 75% of the content in LLM-generated summary is hallucinated, with hallucinations more likely to occur towards the end of the summaries. Moreover, when summarizing non-existent topic-related information, gpt-3.5-turbo and GPT-4o still generate summaries about 79.35% and 44% of the time, raising concerns about their tendency to fabricate content. To understand the characteristics of these hallucinations, we manually evaluate 700+ insights and find that most errors stem from either failing to follow instructions or producing overly generic insights. Motivated by these observations, we investigate the efficacy of simple post-hoc baselines in mitigating hallucinations but find them only moderately effective. Our results underscore the need for more effective approaches to systematically mitigate hallucinations in MDS. We release our dataset and code at github.com/megagonlabs/Hallucination_MDS.

Figures (24)

• Figure 1: An illustrative example of a summary generation from news articles. Concerned about the credibility of the information, a human instructs the model to focus on shared, subtopic-related information. However, the LLM summarizes unrelated information that is not shared, raising concerns about the trustworthiness of LLMs in MDS.
• Figure 2: Performance metrics as a function of input documents counts in the "subtopic" setting. Each line represents the mean value, with shaded areas indicating the 95% confidence intervals. Generally, recall drops significantly as document count increases, while average error rate changes only slightly across models and domains.
• Figure 3: Performance metrics as a function of the number of input documents in the "subtopic+trustworthy" setting. Each line represents the mean value, with shaded areas indicating the 95% confidence intervals.
• Figure 4: Mean and $95$% confidence intervals of summary refusal rate (%) for the news (top) and conversation (bottom) domains. Notably, OpenAI models perform the worst, while Llama 3.1 (70B) consistently abstains from generating summaries over 60+% of the time, regardless of document count.
• Figure 5: Likelihood of an insight coming from a document (y-axis) based on its position in the input (when summarizing 10 documents). GPT-3.5-Turbo, Llama 3.1 (70B) and Gemini (Flash) mistakes seem to be more likely to originate from later documents on average than from earlier ones.