SEAM: A Stochastic Benchmark for Multi-Document Tasks

TL;DR

SEAM introduces a stochastic benchmark to evaluate large language models on multi-document tasks, addressing the lack of a unified, robust evaluation framework for real-world document collections. It evaluates six datasets across three task families—multi-document summarization, multi-hop QA, and cross-document coreference—within a fixed retrieval setting and a highly randomized prompt design. The approach reveals that multi-document reasoning remains challenging for state-of-the-art LLMs, and that results are highly sensitive to prompt variations, which the stochastic framework helps quantify. SEAM thus provides a standardized, repeatable evaluation method and offers insights into improving robustness and formatting adherence in multi-document reasoning systems.

Abstract

Various tasks, such as summarization, multi-hop question answering, or coreference resolution, are naturally phrased over collections of real-world documents. Such tasks present a unique set of challenges, revolving around the lack of coherent narrative structure across documents, which often leads to contradiction, omission, or repetition of information. Despite their real-world application and challenging properties, there is currently no benchmark which specifically measures the abilities of large language models (LLMs) on multi-document tasks. To bridge this gap, we present SEAM (a Stochastic Evaluation Approach for Multi-document tasks), a conglomerate benchmark over a diverse set of multi-document datasets, setting conventional evaluation criteria, input-output formats, and evaluation protocols. In particular, SEAM addresses the sensitivity of LLMs to minor prompt variations through repeated evaluations, where in each evaluation we sample uniformly at random the values of arbitrary factors (e.g., the order of documents). We evaluate different LLMs on SEAM finding that multi-document tasks pose a significant challenge for LLMs, even for state-of-the-art models with 70B parameters. In addition, we show that the stochastic approach uncovers underlying statistical trends which cannot be observed in a static benchmark. We hope that SEAM will spur progress via consistent and meaningful evaluation of multi-document tasks.

Figures (5)

• Figure 1: The Seam benchmark consists of datasets addressing various multi-document processing tasks. An LLM is evaluated against different samplings of prompt configurations, producing results that quantify models' performance on multi-document tasks, as well as their sensitivity to prompt and instance variations.
• Figure 2: The multi-document prompt template used for Seam. Elements in brackets represent template elements filled according to the specific task, dataset, and sample.
• Figure 3: Result distributions across different tasks. Each box-plot shows the distribution of 10 different resamplings for each model, on each task.
• Figure 4: The averaged rank ($AR$) and relative standard deviation ($ARSD$) based on the mean score of all executions, averaged over all datasets. Lower $ARSD$ indicates higher robustness to prompt variations. Lower $AR$ indicates more accurate performance relative to other models.
• Figure 5: Performance as a function of input length.