Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents

Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Yurong Wu, Chenhao Ma, Jian-Guang Lou, Reynold Cheng

TL;DR

Tapilot-Crossing introduces a scalable interactive data-analysis benchmark built via a four-agent Decision Company environment, enabling 1024 interaction logs across Normal, Action, Private, and Private Action modes. It systematically evaluates LLM agents on code and analysis tasks, and proposes Adaptive Interaction Reflection (AIR), a self-generated reflection mechanism that learns from successful histories. Across multiple models, AIR demonstrates notable gains (up to 44.5% relative) and reveals essential insights about long-context handling, private-library usage, and instruction-following in interactive settings. The work highlights the need for efficient, history-aware reasoning in data analysis agents and provides a practical, cost-effective framework for benchmarking and evolving interactive data analysis capabilities.

Abstract

Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%.

Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents

TL;DR

Tapilot-Crossing introduces a scalable interactive data-analysis benchmark built via a four-agent Decision Company environment, enabling 1024 interaction logs across Normal, Action, Private, and Private Action modes. It systematically evaluates LLM agents on code and analysis tasks, and proposes Adaptive Interaction Reflection (AIR), a self-generated reflection mechanism that learns from successful histories. Across multiple models, AIR demonstrates notable gains (up to 44.5% relative) and reveals essential insights about long-context handling, private-library usage, and instruction-following in interactive settings. The work highlights the need for efficient, history-aware reasoning in data analysis agents and provides a practical, cost-effective framework for benchmarking and evolving interactive data analysis capabilities.

Abstract

Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%.
Paper Structure (107 sections, 4 equations, 20 figures, 3 tables)

This paper contains 107 sections, 4 equations, 20 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Interactive Data analysis
  4. Agent Actions in Interactions
  5. Tapilot-Crossing Dataset
  6. Dataset Construction.
  7. Data Acquisition & Prepossessing.
  8. Client Persona Generation.
  9. Simulation of Analysis Scenarios.
  10. Plan Discussion.
  11. Interaction Log Annotation.
  12. Human Calibration
  13. Private Lib Mode Evolution
  14. Data Statistics & Metrics
  15. Dataset Statistics
  16. ...and 92 more sections

Figures (20)

  • Figure 1: This is an overview of the four interaction modes in Tapilot-Crossing. Notable Opponents is ambiguous which requires clarification in multi-turn interaction.
  • Figure 2: This figure provides an overview of action types in Tapilot-Crossing, illustrated by examples. We emphasize the keywords specific to each category, and demonstrate the relevant sections of the associated queries, as well as the agent actions. The number of symbols represents the relative difficulty of each action.
  • Figure 3: This describes the construction pipeline of Tapilot-Crossing by the AI Agent Sandbox Decision Company. denotes human intervention during construction. For a more detailed describing, please refer to Section \ref{['sec:dataset']}.
  • Figure 4: Visualization of the performance of GPT-4-32k across various categories in Action Mode. It includes a comparative analysis of the base, agent, and inter_agent versions.
  • Figure 5:
  • ...and 15 more figures