TiInsight: A SQL-based Automated Exploratory Data Analysis System through Large Language Models

TL;DR

TiInsight tackles cross-domain exploratory data analysis by integrating hierarchical data context (HDC) generation with a text-to-SQL pipeline (TiSQL) and a rule-based visualization component (TiChart) inside a user-friendly GUI. The system employs a two-stage schema filtering and a map-reduce style workflow to map natural-language questions to accurate SQL, followed by an explain–refine–execute self-refinement loop to correct errors. Key innovations include parallelized HDC generation to handle large schemas, coarse-to-fine schema linking, and a perturbation-resistant visualization strategy, all demonstrated in PingCAP's production environment on representative datasets. The work demonstrates practical viability of automated cross-domain EDA, reducing reliance on expert SQL knowledge and enabling end-to-end data exploration with interpretable visual outputs.

Abstract

The SQL-based exploratory data analysis has garnered significant attention within the data analysis community. The emergence of large language models (LLMs) has facilitated the paradigm shift from manual to automated data exploration. However, existing methods generally lack the ability for cross-domain analysis, and the exploration of LLMs capabilities remains insufficient. This paper presents TiInsight, an SQL-based automated cross-domain exploratory data analysis system. First, TiInsight offers a user-friendly GUI enabling users to explore data using natural language queries. Second, TiInsight offers a robust cross-domain exploratory data analysis pipeline: hierarchical data context (i.e., HDC) generation, question clarification and decomposition, text-to-SQL (i.e., TiSQL), and data visualization (i.e., TiChart). Third, we have implemented and deployed TiInsight in the production environment of PingCAP and demonstrated its capabilities using representative datasets. The demo video is available at https://youtu.be/JzYFyYd-emI.

Figures (5)

• Figure 1: Overall architecture of TiInsight.
• Figure 2: Overview of hierarchical data context.
• Figure 3: An illustration of a GUI interface.
• Figure 4: An illustration of an EDA task on a financial dataset.
• Figure 5: An illustration of an EDA result on a financial dataset.