Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design

TL;DR

The findings revealed that ASA-GPT-4o achieved near-flawless execution on designated research missions, underscoring the potential of methods like ASA to achieve automation in simulation research processes to enhance research efficiency.

Abstract

The advent of Large Language Models (LLMs) has created new opportunities for the automation of scientific research spanning both experimental processes and computational simulations. This study explores the feasibility of constructing an autonomous simulation agent (ASA) powered by LLMs through prompt engineering and automated program design to automate the entire simulation research process according to a human-provided research plan. This process includes experimental design, remote upload and simulation execution, data analysis, and report compilation. Using a well-studied simulation problem of polymer chain conformations as a test case, we assessed the long-task completion and reliability of ASAs powered by different LLMs, including GPT-4o, Claude-3.5, etc. Our findings revealed that ASA-GPT-4o achieved near-flawless execution on designated research missions, underscoring the potential of methods like ASA to achieve automation in simulation research processes to enhance research efficiency. The outlined automation can be iteratively performed for up to 20 cycles without human intervention, illustrating the potential of ASA for long-task workflow automation. Additionally, we discussed the intrinsic traits of ASA in managing extensive tasks, focusing on self-validation mechanisms, and the balance between local attention and global oversight.

Figures (4)

• Figure 1: (A) Schematic diagram of the AI automation process for theoretical simulation research conducted in this study. The human researcher writes the research plan (RP) for the theoretical simulation and provides it to the autonomous simulation agent (ASA), which is developed by prompt engineering and automated program design (see the Methods section). The ASA first generates the simulation program and then uploads it to a remote server, performs simulation calculations with different parameter conditions, collects data, and finally writes a report based on the simulation results. (B) Diagram of the ASA dialogue. The human researcher only inputs the RP at the beginning, and then, the ASA makes actions based on the given RP and dialogue history, achieving an automated simulation research process. (C) Two simulation problems are used to test the ASA method in this paper: the chain conformation simulation of a random walk and the gravitational model simulation of asteroid orbit prediction (see the SI).
• Figure 2: Overview of RP 1-3. We designed three RPs related to polymer chain simulation, each containing multiple steps such as simulation, plotting, and report writing, and provided them to the ASA. RP 1 and RP 3 required the ASA to generate a complete Python program to simulate polymer chains, run the simulation locally, and produce a final report. RP 2 required the ASA to modify the provided simulation program, run the simulation on a remote server, and then generate a report. The provided simulation program (omitted in the figure) and remote server information were included in RP 2.
• Figure 3: Completion Rates of Various ASAs for RP 1-3.(A) Table of LLM models tested in this study.(B) Statistics of the number of criteria met by each ASA over 20 experiments. We established seven criteria for each RP to measure mission completion rates and counted the number of times each ASA met these criteria across 20 experiments. Some zero results are not displayed. (C) ASA Scoring. Relative scores for each ASA in RP 1-3 were calculated using the EWM and TOPSIS methods (see the Methods section). (D) ASA Generated Research Report Example. Demonstrates screen cuts for a Word report generated by ASA-GPT-4o for RP 1, including four sections, chain conformation diagrams and a scaling relation fit plot ( see the full text in the SI).
• Figure 4: Recommended practices for automated simulation research workflow. By conducting iterative dialogues with the LLM adhering to a specific design logic, the success rate and efficiency of the automated process are enhanced. The diagrams on the left and right illustrate scenarios without and with the implementation of these strategies, respectively.