SimuAgent: An LLM-Based Simulink Modeling Assistant Enhanced with Reinforcement Learning

TL;DR

SimuAgent tackles the challenge of applying LLMs to graphical, hierarchical Simulink modeling by introducing a compact Python dictionary representation and a lean plan–execute workflow. It embeds three innovations—Staged Training, Abstract–Reconstruct data augmentation, and Reflection-GRPO—to address data sparsity and long-horizon reasoning, achieving fast convergence and robust generalization. On the SimuBench benchmark, the full pipeline with a 7B-Qwen model delivers higher modeling accuracy and faster learning than standard RL baselines and closely approaches GPT-4o with XML prompts, with ablations confirming the value of the two-stage curriculum and ReGRPO. The approach runs entirely on-premise, enabling privacy-preserving, cost-effective AI-assisted engineering and showing promising cross-platform transfer to Modelica and PSCAD.

Abstract

Large language models (LLMs) have revolutionized text-based code automation, but their potential in graph-oriented engineering workflows remains under-explored. We introduce SimuAgent, an LLM-powered modeling and simulation agent tailored for Simulink. SimuAgent replaces verbose XML with a concise, dictionary-style Python representation, dramatically cutting token counts, improving interpretability, and enabling fast, in-process simulation. A lightweight plan-execute architecture, trained in two stages, equips the agent with both low-level tool skills and high-level design reasoning. To tackle sparse rewards in long-horizon tasks, we propose Reflection-GRPO (ReGRPO), which augments Group Relative Policy Optimization (GRPO) with self-reflection traces that supply rich intermediate feedback, accelerating convergence and boosting robustness. Experiments on SimuBench, our newly released benchmark comprising 5300 multi-domain modeling tasks, show that a Qwen2.5-7B model fine-tuned with SimuAgent converges faster and achieves higher modeling accuracy than standard RL baselines, and even surpasses GPT-4o when evaluated with few-shot prompting on the same benchmark. Ablations confirm that the two-stage curriculum and abstract-reconstruct data augmentation further enhance generalization. SimuAgent trains and runs entirely on-premise with modest hardware, delivering a privacy-preserving, cost-effective solution for industrial model-driven engineering. SimuAgent bridges the gap between LLMs and graphical modeling environments, offering a practical solution for AI-assisted engineering design in industrial settings.

Figures (5)

• Figure 1: Comparison between SimuAgent and conventional workflows.
• Figure 2: ReGRPO architecture. Black solid arrows show the original GRPO pipeline. The green dashed lines and boxes highlight ReGRPO’s extensions: the group is split into two subgroups---after Subgroup 1 completes rollouts, reflections on failed rollouts are generated and shared with Subgroup 2 to guide its policy.
• Figure 3: Comprehensive ablation study of ReGRPO training dynamics on the GSM8K dataset. (a) With tool calls disabled, ReGRPO achieves higher returns earlier than vanilla GRPO but shows mild late-stage oscillations; GRPO learns more slowly yet remains stable. (b) The number of self-reflections per episode rapidly diminishes as the agent gains task proficiency, signalling reduced deliberation overhead. (c) When tool invocations are permitted, ReGRPO again exhibits superior sample-efficiency, though both methods eventually plateau at comparable performance levels. (d) Over the course of training, the agent gradually substitutes costly self-reflection with a moderate, steady rate of tool usage, converging to a lean and effective decision-making routine. The shaded error bands represent the variance across six runs with different random seeds.
• Figure 4: Two-stage training analysis on the SimuBench dataset. (a) During Stage 1, ReGRPO learns markedly faster than vanilla GRPO and attains a higher asymptotic reward. (b) Tracing Stage 1 behaviour shows an early surge in both reflections and tool calls; as competence grows, reflections are pruned while a modest, steady tool-invocation rate is retained. (c) In Stage 2, injecting abstract–reconstruct augmented data accelerates convergence and raises the final reward, indicating improved robustness and generalization. (d) Skipping Stage 1 and training Stage 2 from scratch yields noticeably slower learning and a lower plateau, highlighting the benefit of the curriculum.
• Figure 5: Runtime comparison between GRPO and ReGRPO. (a) Mean wall‐clock duration per rollout (no‐tool setting). ReGRPO is consistently slower than GRPO, yet both methods exhibit similar scaling with the number of turns. (b) Percentage runtime overhead of ReGRPO relative to GRPO. Although ReGRPO incurs a noticeable slowdown for short rollouts, the relative overhead steadily diminishes—from about 13.5% at one turn to about 7.2% at eight turns—indicating that its extra cost becomes less significant as rollout length (and thus typical tool‐calling scenarios) increases.