MADE: Benchmark Environments for Closed-Loop Materials Discovery

TL;DR

MADE recasts materials discovery as a closed-loop, end-to-end sequential decision problem, formalizing stability relative to the convex hull with $\Delta_{\text{hull}}(s,H_t)\le\epsilon$ and optimizing for new stable discoveries under a budget $B$. The framework is modular and configurable, enabling end-to-end pipelines or fully agentic systems that orchestrate generators, planners, filters, and scorers, with discovery-centric metrics such as AUDC, AF, and EF to quantify efficiency and diversity. Empirical results across ternary to quinary systems show that learned generators and MLIP surrogate ranking accelerate simple tasks, while planning and agentic orchestration become increasingly valuable as search spaces grow or surrogate accuracy declines; end-to-end agentic policies can rival optimized modular pipelines. MADE thus provides a scalable, reproducible testbed for advancing autonomous scientific discovery, with the potential to inform future RL-enabled, multi-objective materials exploration.

Abstract

Existing benchmarks for computational materials discovery primarily evaluate static predictive tasks or isolated computational sub-tasks. While valuable, these evaluations neglect the inherently iterative and adaptive nature of scientific discovery. We introduce MAterials Discovery Environments (MADE), a novel framework for benchmarking end-to-end autonomous materials discovery pipelines. MADE simulates closed-loop discovery campaigns in which an agent or algorithm proposes, evaluates, and refines candidate materials under a constrained oracle budget, capturing the sequential and resource-limited nature of real discovery workflows. We formalize discovery as a search for thermodynamically stable compounds relative to a given convex hull, and evaluate efficacy and efficiency via comparison to baseline algorithms. The framework is flexible; users can compose discovery agents from interchangeable components such as generative models, filters, and planners, enabling the study of arbitrary workflows ranging from fixed pipelines to fully agentic systems with tool use and adaptive decision making. We demonstrate this by conducting systematic experiments across a family of systems, enabling ablation of components in discovery pipelines, and comparison of how methods scale with system complexity.

Figures (15)

• Figure 1: Example acquisition curves for different discovery policies on two quinary inter-metallic systems. The acceleration factor (AF) and enhancement factor (EF) are shown for the LLM orchestrator policy with respect to a random generator baseline policy. Shaded regions are standard error in the mean across 5 episodes.
• Figure 2: Conceptual overview of the MADE benchmark. a) Existing discovery pipelines and benchmarks follow a static filtering process, moving sequentially from generative models to increasingly expensive evaluation methods, without end-to-end feedback. b) MADE simulates a closed-loop discovery environment where agents iteratively propose candidates, receive oracle feedback (formation energy), and update their strategy. c) Modular, extensible components of the benchmark environments.
• Figure 3: End-to-end materials discovery performance of different policies averaged across all system sizes and episodes, against a random generator baseline. Error bars are one standard error in the mean. See Section \ref{['sec:results']} for details on experimental setup.
• Figure 4: Performance of policies at increasing system size. Shaded regions are standard error in the mean across 10 systems with 5 episodes each. We see larger gains for effective planning on larger search spaces over baselines.
• Figure 5: Performance of policies at varying stability threshold for discovery. Shaded regions are standard error in the mean across 10 systems with 5 episodes each. Surrogate model ranking (MLIPs) do not generalise well to smaller tolerances due to errors, whereas planning algorithms lead to significant gains over baselines.