SUMIE: A Synthetic Benchmark for Incremental Entity Summarization

TL;DR

SUMIE introduces a fully synthetic benchmark for Incremental Entity Summarization (IES), designed to stress-test LLMs on updating and maintaining accurate entity summaries as information evolves. The dataset generation pipeline produces diverse entities, attributes, incremental updates, aligned paragraphs with citations, and distractors to mimic real-world data; the authors also define rigorous alignment and evaluation protocols. Baseline experiments with Update and Merge show current LLMs struggle to exceed an F1 of 80.4%, with MERGE generally outperforming UPDATE; distractors significantly degrade performance and human evaluations reveal notable redundancy and hallucination issues. By open-sourcing SUMIE and its evaluation metrics, the work aims to accelerate progress in robust, evidence-grounded incremental summarization for up-to-date knowledge bases and search systems.

Abstract

No existing dataset adequately tests how well language models can incrementally update entity summaries - a crucial ability as these models rapidly advance. The Incremental Entity Summarization (IES) task is vital for maintaining accurate, up-to-date knowledge. To address this, we introduce SUMIE, a fully synthetic dataset designed to expose real-world IES challenges. This dataset effectively highlights problems like incorrect entity association and incomplete information presentation. Unlike common synthetic datasets, ours captures the complexity and nuances found in real-world data. We generate informative and diverse attributes, summaries, and unstructured paragraphs in sequence, ensuring high quality. The alignment between generated summaries and paragraphs exceeds 96%, confirming the dataset's quality. Extensive experiments demonstrate the dataset's difficulty - state-of-the-art LLMs struggle to update summaries with an F1 higher than 80.4%. We will open source the benchmark and the evaluation metrics to help the community make progress on IES tasks.

Figures (24)

• Figure 1: Overview of the Incremental Entity Summarization Task. Existing attribute ("Impression") can be updated and new attribute ("Camera") can be augmented.
• Figure 2: Dataset Generation Methodology Overview: (1) Generate entity names (masked for ethics consideration) and attributes. (2) Create default summary table with diverse values. (3) Sample attributes/values for incremental summaries (* repeated sampling, ** conflicting values). (4) Generate paragraphs with varying tones based on attributes/values. (5) Verify summary table/paragraph alignment. (6) Add distractor sentence. Note that attribute values and sentences in the same color should be aligned and bold texts in paragraphs are the evidences for corresponding attribute values.
• Figure 3: F1 scores across 10 categories (see Appendix \ref{['app:performance-chart']} for the rest.).
• Figure 4: An example of an LLM distracted by irrelevant information.
• Figure 5: Generate Attribute Instruction.