ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows

TL;DR

ML-Dev-Bench introduces a comprehensive, end-to-end benchmark for evaluating AI agents on real-world ML development workflows across 30 tasks. Using the Calipers framework, it assesses ReAct, Openhands, and AIDE on dataset handling, training, debugging, implementation, and API integration, revealing Openhands-Sonnet as the strongest performer with a 50% success rate. The results highlight that agent performance deteriorates on open-ended, long-running tasks and reveal distinct failure modes across agents, underscoring the current limits of automation in practical ML development. The work provides an open-source framework for community-driven extension and benchmarking, with clear directions for scaling compute, variance analysis, and exploration of new reasoning models.

Abstract

In this report, we present ML-Dev-Bench, a benchmark aimed at testing agentic capabilities on applied Machine Learning development tasks. While existing benchmarks focus on isolated coding tasks or Kaggle-style competitions, ML-Dev-Bench tests agents' ability to handle the full complexity of ML development workflows. The benchmark assesses performance across critical aspects including dataset handling, model training, improving existing models, debugging, and API integration with popular ML tools. We evaluate three agents - ReAct, Openhands, and AIDE - on a diverse set of 30 tasks, providing insights into their strengths and limitations in handling practical ML development challenges. We open source the benchmark for the benefit of the community at \href{https://github.com/ml-dev-bench/ml-dev-bench}{https://github.com/ml-dev-bench/ml-dev-bench}.