Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

TL;DR

This paper tackles the challenge of steering and verifying AI-assisted data analysis with large language models by introducing two interactive task-decomposition interfaces, Phasewise and Stepwise. Through a formative study and a controlled within-subjects experiment, the authors show that exposing editable AI assumptions, structured progression, and side conversations yields greater perceived control and easier verification compared with a standard conversational baseline. The work provides design guidelines for AI-assisted data analysis tools, highlighting trade-offs between information overload and control, and demonstrates the value of progressive disclosure and co-audit capabilities for reliable data analysis workflows. These insights have practical impact for building more trustworthy, auditable, and user-driven AI data-analysis assistants in real-world settings.

Abstract

LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools.

Figures (8)

• Figure 1: The starting input for all systems, which includes a button to upload datasets , selectable datasets to be included in the analysis , and a text input for entering the natural language description that specifies the data analysis task .
• Figure 2: Overview of the Phasewise system's task flow, which decomposes tasks into three stages. Input + Output Assumptions, allows users to upload a dataset , manage column-based AI assumptions , inspect column-based descriptive statistics , add columns missed by the AI , and edit assumptions about the task's output . Execution Plan contains the AI's editable natural language plan for solving the task , which includes user-selectable optional steps . Code and Output contains AI generated code for solving the task and includes an code editor , intermediary variable inspector , and the code output . Section \ref{['sec:taskdecompstruc']} details these features.
• Figure 3: Overview of the Stepwise system's task flow, which decomposes each task into subgoals containing two components. Assumptions and Actions includes the NL subgoal and editable AI assumptions . Code and Output contains a code editor , a dataframe and variable inspector , and the code output . Section \ref{['sec:taskdecompincremental']} details these features.
• Figure 4: The tabbed ribbon displays all the branches created after editing various nodes. Users can select a different tab to switch to that branch. Each tab indicates where the edit occurred and how much it has progressed (number of total nodes).
• Figure 5: In the Stepwise and Phasewise systems, users can select any code in the editor to ask questions from the AI. This will create a question box to the right of the main components in which users can ask their clarification question . The question box will then be replaced with the AI's response , based on the query and the selected text.