ConDABench: Interactive Evaluation of Language Models for Data Analysis

TL;DR

ConDABench introduces a modular, code-grounded framework to generate and evaluate Conversational Data Analysis (ConDA) benchmarks. It combines a multi-agent bench-creation pipeline (curation and code-generation) with an interaction-focused evaluation harness that uses a User Proxy to simulate realistic data-analysis conversations and measure both correctness and conversational quality. The dataset comprises 1,420 ConDA problems grounded in 338 real-world articles, sourced from diverse venues, and validated through automated and human checks. Results show that while recent LLM generations improve problem-solving rates, they struggle with sustaining long, interactive collaboration, highlighting the need for true cooperative AI in data analysis and providing a robust platform for ongoing progress benchmarking.

Abstract

Real-world data analysis tasks often come with under-specified goals and unclean data. User interaction is necessary to understand and disambiguate a user's intent, and hence, essential to solving these complex tasks. Existing benchmarks for evaluating LLMs on data analysis tasks do not capture these complexities or provide first-class support for interactivity. We introduce ConDABench, a framework for generating conversational data analysis (ConDA) benchmarks and evaluating external tools on the generated benchmarks. \bench consists of (a) a multi-agent workflow for generating realistic benchmarks from articles describing insights gained from public datasets, (b) 1,420 ConDA problems generated using this workflow, and (c) an evaluation harness that, for the first time, makes it possible to systematically evaluate conversational data analysis tools on the generated ConDA problems. Evaluation of state-of-the-art LLMs on the benchmarks reveals that while the new generation of models are better at solving more instances, they are not necessarily better at solving tasks that require sustained, long-form engagement. ConDABench is an avenue for model builders to measure progress towards truly collaborative models that can complete complex interactive tasks.

Figures (21)

• Figure 1: ConDABench Generation. From an article and data $d$, Curation pipeline extracts a query-answer pair $(q,a)$ for which the code generation pipeline produces code $c$ to support answer $a$. Details on agents’ implementation and prompt outlines can be found on https://github.com/condabench/condabench_details.
• Figure 2: Running Example and a tree of conversation trajectories
• Figure 3: Code Generator-Reviewer.
• Figure 4: Evaluation Framework.
• Figure 5: Performance vs Conversation Quality (Score vs ConvQ) Across Frameworks. Each point is a model–framework pair (squares: Assistants API; circles: InfiAgent-Conv; triangles: Tool Call). GPT-4o is not a contestant in our evaluations; it serves only as a fixed reference point. The dashed red and blue lines mark GPT-4o’s average Score ($58.6 \%$) and ConvQ ($76.1 \%$), respectively. Models in the upper-right quadrant exceed this GPT-4o reference on both correctness and conversation quality, indicating they reach the correct solution while maintaining high conversational quality.