TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents

TL;DR

TimeSeriesGym tackles the lack of scalable, holistic AI-agent benchmarks for ML engineering tasks by providing a time-series focused, open-source benchmark with 34 challenges across 8 problem types from 15 domains; it supports multimodal evaluation of artifacts (data, code, models) and scalable challenge generation, including a Lite subset for cost-effective testing. The framework is agent-agnostic and uses a hybrid evaluation combining objective metrics with LLM-based qualitative judgments to identify skill gaps and guide improvement. Experiments show that while advanced scaffolds like AIDE paired with reasoning LLMs yield more valid submissions, many TimeSeriesGym challenges remain hard for state-of-the-art agents and increasing time/resources yields limited gains. The work highlights important issues like data leakage, plagiarism, resource allocation, and societal impact, and open-sources the toolkit for reproducibility and future development.

Abstract

We introduce TimeSeriesGym, a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents on time series machine learning engineering challenges. Existing benchmarks lack scalability, focus narrowly on model building in well-defined settings, and evaluate only a limited set of research artifacts (e.g., CSV submission files). To make AI agent benchmarking more relevant to the practice of machine learning engineering, our framework scales along two critical dimensions. First, recognizing that effective ML engineering requires a range of diverse skills, TimeSeriesGym incorporates challenges from diverse sources spanning multiple domains and tasks. We design challenges to evaluate both isolated capabilities (including data handling, understanding research repositories, and code translation) and their combinations, and rather than addressing each challenge independently, we develop tools that support designing multiple challenges at scale. Second, we implement evaluation mechanisms for multiple research artifacts, including submission files, code, and models, using both precise numeric measures and more flexible LLM-based evaluation approaches. This dual strategy balances objective assessment with contextual judgment. Although our initial focus is on time series applications, our framework can be readily extended to other data modalities, broadly enhancing the comprehensiveness and practical utility of agentic AI evaluation. We open-source our benchmarking framework to facilitate future research on the ML engineering capabilities of AI agents.

Figures (6)

• Figure 1: TimeSeriesGym is a scalable benchmarking environment for ML engineering agents. It currently features 34 time series challenges across 8 unique time series problems, spanning more than 15 domains. Challenges are either carefully designed based on real-world ML practice, or sourced from Kaggle competitions and GitHub repositories. TimeSeriesGym includes key mechanisms to enable efficient and scalable generation of new challenges. Our evaluation methodology combines precise quantitative metrics with flexible qualitative assessment, and provides specialized tools to grade various artifacts generated during ML engineering. TimeSeriesGym is compatible with many different agent types, even those with fundamentally distinct designs.
• Figure 2: GPT-4.1's familiarity with TimeSeriesGym challenges, compared to its familiarity with MLE-bench.
• Figure 3: The prompt we use to initialize all scaffolds, adapted from MLE-benchchan2025mlebench.
• Figure 4: OpenHands wastes 5 steps on inspecting model file while the correct way to import the model is in README
• Figure 5: AIDE’s interpreter does not execute code under main environment