AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions

TL;DR

AutoKaggle tackles the challenge of automating end-to-end tabular data science pipelines by combining a phase-based workflow with a collaborative multi-agent system. It introduces five specialized agents and a comprehensive tool library, backed by iterative debugging and unit testing to ensure code quality and transparency. The framework is evaluated on eight Kaggle competitions, achieving a validation submission rate of 0.85 and a comprehensive score of 0.82, demonstrating practical effectiveness. This work advances AI-assisted data science by integrating planning, execution, testing, and reporting into an interpretable, user-controllable automation platform.

Abstract

Data science tasks involving tabular data present complex challenges that require sophisticated problem-solving approaches. We propose AutoKaggle, a powerful and user-centric framework that assists data scientists in completing daily data pipelines through a collaborative multi-agent system. AutoKaggle implements an iterative development process that combines code execution, debugging, and comprehensive unit testing to ensure code correctness and logic consistency. The framework offers highly customizable workflows, allowing users to intervene at each phase, thus integrating automated intelligence with human expertise. Our universal data science toolkit, comprising validated functions for data cleaning, feature engineering, and modeling, forms the foundation of this solution, enhancing productivity by streamlining common tasks. We selected 8 Kaggle competitions to simulate data processing workflows in real-world application scenarios. Evaluation results demonstrate that AutoKaggle achieves a validation submission rate of 0.85 and a comprehensive score of 0.82 in typical data science pipelines, fully proving its effectiveness and practicality in handling complex data science tasks.

Figures (8)

• Figure 1: Overview of AutoKaggle. AutoKaggle integrates a phase-based workflow with specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer), iterative debugging and unit testing, a comprehensive machine learning tools library, and detailed reporting.
• Figure 2: Iterative debugging and testing.
• Figure 3: Average normalized performance score for different settings/tasks.
• Figure 4: Left. Debugging time and Right. Average performance in competitions.
• Figure 5: Comprehensive Score across different debugging times.