MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering

TL;DR

MLE-Dojo addresses the need for interactive, executable benchmarking of autonomous LLM agents in machine learning engineering by introducing a Gym-style environment built on 200+ Kaggle tasks. It provides a modular, Dockerized task space with standardized observations, a minimal action set centered on Python code, and a HumanRank-based reward that enables iterative training via supervised fine-tuning and reinforcement learning. Extensive evaluations across eight frontier LLMs reveal meaningful iterative improvements but underscore limitations in long-horizon reasoning and robust error handling, highlighting key gaps for future research. By open-sourcing the framework and maintaining a public leaderboard, MLE-Dojo aims to accelerate reproducible, scalable development of next-generation MLE agents.

Abstract

We introduce MLE-Dojo, a Gym-style framework for systematically reinforcement learning, evaluating, and improving autonomous large language model (LLM) agents in iterative machine learning engineering (MLE) workflows. Unlike existing benchmarks that primarily rely on static datasets or single-attempt evaluations, MLE-Dojo provides an interactive environment enabling agents to iteratively experiment, debug, and refine solutions through structured feedback loops. Built upon 200+ real-world Kaggle challenges, MLE-Dojo covers diverse, open-ended MLE tasks carefully curated to reflect realistic engineering scenarios such as data processing, architecture search, hyperparameter tuning, and code debugging. Its fully executable environment supports comprehensive agent training via both supervised fine-tuning and reinforcement learning, facilitating iterative experimentation, realistic data sampling, and real-time outcome verification. Extensive evaluations of eight frontier LLMs reveal that while current models achieve meaningful iterative improvements, they still exhibit significant limitations in autonomously generating long-horizon solutions and efficiently resolving complex errors. Furthermore, MLE-Dojo's flexible and extensible architecture seamlessly integrates diverse data sources, tools, and evaluation protocols, uniquely enabling model-based agent tuning and promoting interoperability, scalability, and reproducibility. We open-source our framework and benchmarks to foster community-driven innovation towards next-generation MLE agents.

Figures (16)

• Figure 1: Benchmark evaluations of eight frontiers LLMs across 50 evaluation tasks in MLE-Dojo.
• Figure 2: Overview of task diversity in MLE-Dojo, highlighting representative examples from four major domains: time series, computer vision, tabular data, and natural language processing.
• Figure 3: Overview of data structure in MLE-Dojo.
• Figure 4: Overview of MLE-Dojo. The framework bridges MLE-Agents with MLE task environments through standardized interfaces for observation and action spaces.
• Figure 5: Interaction loop in MLE-Dojo with theoretical model (left) and concrete Python API (right).