An LLM-Based Approach for Insight Generation in Data Analysis

TL;DR

The paper presents an LLM-driven framework for extracting insightful, actionable textual insights from multi-table databases. It combines a Hypothesis Generator to formulate high-level questions, a Query Agent to generate and validate SQL queries, and a Summarization module to verbalize results into concise insights, with iterative hallucination mitigation. Insights are evaluated via a hybrid human–LLM scheme, using an Elo-based ranking for insightfulness and a truth-value-based measure for correctness, demonstrating improvements over baselines on both private and public datasets. The approach emphasizes scalability, cost-efficiency, and robust assessment, with potential impact across business analytics, healthcare, and research domains.

Abstract

Generating insightful and actionable information from databases is critical in data analysis. This paper introduces a novel approach using Large Language Models (LLMs) to automatically generate textual insights. Given a multi-table database as input, our method leverages LLMs to produce concise, text-based insights that reflect interesting patterns in the tables. Our framework includes a Hypothesis Generator to formulate domain-relevant questions, a Query Agent to answer such questions by generating SQL queries against a database, and a Summarization module to verbalize the insights. The insights are evaluated for both correctness and subjective insightfulness using a hybrid model of human judgment and automated metrics. Experimental results on public and enterprise databases demonstrate that our approach generates more insightful insights than other approaches while maintaining correctness.

Figures (8)

• Figure 1: Overview of our approach: the Hypothesis Generator generates interesting, high-level questions, the Query Agent answers them using SQL scripts, and the Summarization module aggregates the results into an insight.
• Figure 2: Proposed Architecture: The High-Level Generator generates questions using a short description , the Low-Level Generator splits each question into subquestions that are easier to answer by giving all the database details. The Query Agent uses SQL queries to answer those questions and validates them with LLM evaluation. Finally, the Summarizer aggregates the answers into a short insight and iteratively removes hallucinations to generate the final result.
• Figure 3: Human Elo Bootstrap Confidence Intervals (95%) in the database private_sales. Each bootstrap sample has a different order of comparisons.
• Figure 4: LLM Elo Bootstrap Confidence Intervals (95%) in the database private_sales. Each bootstrap sample has a different order of comparisons.
• Figure 5: LLM Bootstrap Confidence Intervals (95%) in public BIRD databases. Each bootstrap sample has a different order of comparisons.