Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents

TL;DR

This work introduces MatDesign, a real-world benchmark of goals and constraints for materials discovery, and AccelMat, a multi-agent LLM framework that generates and iteratively refines material hypotheses via critic feedback and optional knowledge-graph grounding. The dual evaluation metrics—Closeness and Quality—assess both alignment to ground-truth hypotheses and broader methodological validity, novelty, and feasibility. Experimental results show that incorporating both critic feedback and a materials knowledge graph yields the strongest performance, improving consensus among critics and the practicality of proposed hypotheses. The study provides a scalable, human-aligned approach to accelerating materials discovery with large language models, while acknowledging limitations in data size and potential clarity gaps in automated critiques.

Abstract

Materials discovery and design are essential for advancing technology across various industries by enabling the development of application-specific materials. Recent research has leveraged Large Language Models (LLMs) to accelerate this process. We explore the potential of LLMs to generate viable hypotheses that, once validated, can expedite materials discovery. Collaborating with materials science experts, we curated a novel dataset from recent journal publications, featuring real-world goals, constraints, and methods for designing real-world applications. Using this dataset, we test LLM-based agents that generate hypotheses for achieving given goals under specific constraints. To assess the relevance and quality of these hypotheses, we propose a novel scalable evaluation metric that emulates the process a materials scientist would use to evaluate a hypothesis critically. Our curated dataset, proposed method, and evaluation framework aim to advance future research in accelerating materials discovery and design with LLMs.

Figures (3)

• Figure 1: Overview of our iterative hypothesis generation and evaluation pipeline. Starting from an input prompt and a knowledge graph, the Hypotheses Generator (GPT-4o) proposes 20 hypotheses, which are then reviewed by three critics--GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Flash. Their feedback is consolidated by the Summarizer (GPT-4o); if unanimous agreement is not reached, the hypotheses along with critic feedback are fed back to the Hypotheses Generator for refinement and are re-evaluated by the critics. Once approved, the final hypotheses proceed to the Evaluation Agent (OpenAI-o1-preview) for scoring.
• Figure 2: The left plot illustrates the Closeness metric scores across three evaluation criteria for the three configurations. The right plot depicts the Quality metric scores across six evaluation criteria for the same configurations. Both plots highlight that integrating feedback from Critic Agents and leveraging contextual knowledge from the Knowledge Graph enhances performance.
• Figure 3: The left plot compares the performance of open-source and closed-source models on the Closeness metric, while the right plot compares their Quality scores. Both plots clearly show that closed-source models outperform open-source models significantly.