Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework

TL;DR

This work addresses the challenge of enabling LLMs to perform power system simulations by introducing a feedback-driven, multi-agent framework that couples an enhanced retrieval-augmented generation (RAG) module, an advanced reasoning module, and an environmental acting module with an error-feedback loop. The approach decomposes retrieval queries into function- and option-related components, uses a triple-based knowledge base to capture dependencies, and employs few-shot chain-of-thought prompting to guide code generation, all while iteratively executing and correcting simulation tasks. Empirical results on 69 tasks across Daline and MATPOWER show dramatic improvements over baseline LLMs and supervised fine-tuning, achieving up to $96.85\%$ success and enabling cost-effective, rapid simulations (~$0.014$ USD per task). The framework demonstrates strong potential for intelligent, domain-specific LLM assistants in power system research and beyond, while also highlighting areas for future work such as enhanced error checking and interactive clarification to approach 100% reliability.

Abstract

The integration of experimental technologies with large language models (LLMs) is transforming scientific research. It positions AI as a versatile research assistant rather than a mere problem-solving tool. In the field of power systems, however, managing simulations -- one of the essential experimental technologies -- remains a challenge for LLMs due to their limited domain-specific knowledge, restricted reasoning capabilities, and imprecise handling of simulation parameters. To address these limitations, this paper proposes a feedback-driven, multi-agent framework. It incorporates three proposed modules: an enhanced retrieval-augmented generation (RAG) module, an improved reasoning module, and a dynamic environmental acting module with an error-feedback mechanism. Validated on 69 diverse tasks from Daline and MATPOWER, this framework achieves success rates of 93.13% and 96.85%, respectively. It significantly outperforms ChatGPT 4o, o1-preview, and the fine-tuned GPT-4o, which all achieved a success rate lower than 30% on complex tasks. Additionally, the proposed framework also supports rapid, cost-effective task execution, completing each simulation in approximately 30 seconds at an average cost of 0.014 USD for tokens. Overall, this adaptable framework lays a foundation for developing intelligent LLM-based assistants for human researchers, facilitating power system research and beyond.

Figures (16)

• Figure 1: The feedback-driven multi-agent framework. It consists of an enhanced RAG module, an advanced reasoning module, and an environmental acting module, all interconnected through an error-feedback mechanism. This framework enables iterative refinement by incorporating simulation-specific knowledge, improving reasoning for complex simulation tasks, and facilitating environmental interaction to generate accurate simulation results.
• Figure 2: Enhanced RAG module for simulation tasks. (a) The retrieval agent decomposes simulation requests into function-related and option-related sub-queries, mapped to specific functions and options for precise keyword-based retrieval. (b) Structured prompt design detailing keyword extraction steps via few-shot CoT.
• Figure 3: General RAG diagram, including the process of external knowledge chunking, text embedding, and parallel retrieval within a vector database to produce relevant retrieval output based on the input queries and the external knowledge base. Algorithm \ref{['alg:embed']} details the configuration settings and process for the embedding used in this paper.
• Figure 4: Enhanced reasoning module for simulation code generation: (a) Core Concept, defining the coding agent’s role, assigned tasks, reasoning path, and contextual knowledge. (b) Structured prompt design equipped with few-shot CoT, in order to enhance the agent's reasoning ability for simulation. (c) Coding agent workflow integrating the designed prompt and simulation request to produce simulation code.
• Figure 5: Environmental acting module with an error-feedback mechanism that integrates with both the RAG and reasoning modules.