Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

Alexis Audran-Reiss, Jordi Armengol-Estapé, Karen Hambardzumyan, Amar Budhiraja, Martin Josifoski, Edan Toledo, Rishi Hazra, Despoina Magka, Michael Shvartsman, Parth Pathak, Justine T Kao, Lucia Cipolina-Kun, Bhavul Gauri, Jean-Christophe Gagnon-Audet, Emanuel Tewolde, Jenny Zhang, Taco Cohen, Yossi Adi, Tatiana Shavrina, Yoram Bachrach

TL;DR

This study investigates ideation diversity as a key determinant of AI research agents' performance on MLE-bench. By analyzing 11,000 trajectories across 6 LLM backbones and 2 agential scaffolds, the work shows that higher ideation diversity correlates with better performance and provides causal evidence via a controlled diversity-ablating experiment. It introduces a formal measure of diversity, demonstrates scaffold-dependent variability in diversity, and validates findings with alternative performance metrics beyond medal counts. The results highlight the importance of balancing diverse ideation with implementation capabilities and suggest directions for diversity-aware designs in future AI research agents and benchmarks.

Abstract

AI research agents offer the promise to accelerate scientific progress by automating the design, implementation, and training of machine learning models. However, the field is still in its infancy, and the key factors driving the success or failure of agent trajectories are not fully understood. We examine the role that ideation diversity plays in agent performance. First, we analyse agent trajectories on MLE-bench, a well-known benchmark to evaluate AI research agents, across different models and agent scaffolds. Our analysis reveals that different models and agent scaffolds yield varying degrees of ideation diversity, and that higher-performing agents tend to have increased ideation diversity. Further, we run a controlled experiment where we modify the degree of ideation diversity, demonstrating that higher ideation diversity results in stronger performance. Finally, we strengthen our results by examining additional evaluation metrics beyond the standard medal-based scoring of MLE-bench, showing that our findings still hold across other agent performance metrics.

What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

TL;DR

This study investigates ideation diversity as a key determinant of AI research agents' performance on MLE-bench. By analyzing 11,000 trajectories across 6 LLM backbones and 2 agential scaffolds, the work shows that higher ideation diversity correlates with better performance and provides causal evidence via a controlled diversity-ablating experiment. It introduces a formal measure of diversity, demonstrates scaffold-dependent variability in diversity, and validates findings with alternative performance metrics beyond medal counts. The results highlight the importance of balancing diverse ideation with implementation capabilities and suggest directions for diversity-aware designs in future AI research agents and benchmarks.

Abstract

AI research agents offer the promise to accelerate scientific progress by automating the design, implementation, and training of machine learning models. However, the field is still in its infancy, and the key factors driving the success or failure of agent trajectories are not fully understood. We examine the role that ideation diversity plays in agent performance. First, we analyse agent trajectories on MLE-bench, a well-known benchmark to evaluate AI research agents, across different models and agent scaffolds. Our analysis reveals that different models and agent scaffolds yield varying degrees of ideation diversity, and that higher-performing agents tend to have increased ideation diversity. Further, we run a controlled experiment where we modify the degree of ideation diversity, demonstrating that higher ideation diversity results in stronger performance. Finally, we strengthen our results by examining additional evaluation metrics beyond the standard medal-based scoring of MLE-bench, showing that our findings still hold across other agent performance metrics.

Paper Structure

This paper contains 52 sections, 20 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Research Agents and MLE-bench
  4. Agents
  5. MLE-Bench
  6. Methods
  7. General Setup
  8. Dataset
  9. Metrics
  10. Agentic Scaffolds
  11. LLM Backbones
  12. Measuring Ideation Diversity
  13. Controlling Ideation Diversity
  14. Baseline agents
  15. Agents with ablated diversity
  16. ...and 37 more sections

Figures (20)

  • Figure 1: Ideation diversity correlates with performance in MLE-bench: Our analysis shows that ideation diversity correlates with the agent's trajectory success in MLE-bench. To confirm this relation, we later intervene on ideation diversity in a controlled experiment, in Section \ref{['sec:results_controlled_exp']}.
  • Figure 2: An example flow of an AI research agent attempting an MLE-bench task. The goal of the task is to build a multi-headed model to classify different types of toxicity threats. The agent first tries the idea to finetune a model end to end, but the code fails and the agent fixes the bug. After analysis of this approach, the agent continues improving the solution, producing more nodes.
  • Figure 4: Correlation between tree-level diversity and performance on MLE-bench
  • Figure 5: Number of distinct architectures per task - Cumulative Distribution
  • Figure 6: Comparison of MLE-bench lite medal rate of $\text{AIRA}_\text{Greedy}$ and $\text{AIRA}_\text{MCTS}$ with and without interventions to reduce solution diversity (as indicated by '- Low diversity'). Error bars represent 95% confidence intervals computed using stratified bootstrapping, using the rliable library agarwal2021deep.
  • ...and 15 more figures