SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

TL;DR

SUPER tackles reproducibility by introducing an end-to-end benchmark for evaluating LLM-based agents on setting up and executing tasks from research repositories, with Expert, Masked, and Auto problem sets. The authors implement a dual evaluation scheme—exact accuracy against gold solutions and landmark-based progress signals—to capture both final correctness and incremental progress. Experiments show current state-of-the-art models struggle substantially, especially on end-to-end tasks (around 16% accuracy) while still making progress on sub-tasks, highlighting core challenges in repository understanding and code editing. The dataset's low-profile repositories and varied task types provide a challenging, scalable platform for developing and measuring improvements in autonomous research-execution agents, with the Auto set offering a resource for ongoing development and fine-tuning.

Abstract

Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.

Figures (6)

• Figure 1: An illustration of a research task and some of the steps an agent would need to complete it, including updating data loading configuration, resolving dependency issues (due to unlisted version dependencies in the repository), running the training script and reporting metrics.
• Figure 2: An overview of the construction pipeline for the Expert and Masked sets. The Expert set contains manually written tasks, along with expert solutions (Step 2). The Masked set contains problems extracted from the experts set (Step 3).
• Figure 3: An abstract demonstration of how sub-problems are extracted: starting from a gold end-to-end task solution (left), we remove cells (middle) that focus on certain aspects (differently colored cells in the figure), then create a masked problem by defining a goal and prefix cells (right). The prefix cells are executed in the environment, and the agent must then write code to solve the sub-problem.
• Figure 4: Guidelines provided to experts.
• Figure 5: Guidelines provided to experts (continued).