InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents

TL;DR

<3-5 sentence high-level summary> InsightEval addresses the lack of robust benchmarks for insight discovery in LLM-driven data agents by diagnosing flaws in existing datasets, defining stringent benchmark requirements, and introducing a data-curation pipeline that yields 1000 insights across six types. It couples a comprehensive evaluation framework with novel metrics, including Insight F1 and novelty, to quantify both correctness and the discovery of unannotated insights. Empirical results across two agent frameworks and multiple backbones demonstrate that Insight F1 aligns more closely with human judgments and reveal challenges in automated insight discovery such as limited exploratory breadth and backbone-driven novelty. The work offers a practical, extensible benchmark to drive progress in automated data analysis and insight discovery for tabular data.

Abstract

Data analysis has become an indispensable part of scientific research. To discover the latent knowledge and insights hidden within massive datasets, we need to perform deep exploratory analysis to realize their full value. With the advent of large language models (LLMs) and multi-agent systems, more and more researchers are making use of these technologies for insight discovery. However, there are few benchmarks for evaluating insight discovery capabilities. As one of the most comprehensive existing frameworks, InsightBench also suffers from many critical flaws: format inconsistencies, poorly conceived objectives, and redundant insights. These issues may significantly affect the quality of data and the evaluation of agents. To address these issues, we thoroughly investigate shortcomings in InsightBench and propose essential criteria for a high-quality insight benchmark. Regarding this, we develop a data-curation pipeline to construct a new dataset named InsightEval. We further introduce a novel metric to measure the exploratory performance of agents. Through extensive experiments on InsightEval, we highlight prevailing challenges in automated insight discovery and raise some key findings to guide future research in this promising direction.

Figures (17)

• Figure 1: The proportion of each error type. E1, E2, E3, E4, E5 correspond to the Error Type in Table \ref{['InsightBenchIssue']}.
• Figure 2: The dataset construction pipeline of InsightEval. This pipeline consists of 4 steps: 1) Refine the original goal through data description and schema. 2) Construct a new question list by verifying existing questions and generating new questions. 3) Extract key information by code generation and execution, then answer all the questions based on the refined goal, data schema, and extracted data, and finally generate insights. 4) Summarize all the insights by referring to the goal.
• Figure 3: Data Statistics in InsightEval.
• Figure 4: Data Type Distribution and Token Counts
• Figure 5: Comparison of G-Eval scores in Insight Recall and Insight $F1$, and Expert Scores in Human Evaluation.