SimLLM: Fine-Tuning Code LLMs for SimPy-Based Queueing System Simulation

TL;DR

This work tackles the challenge of generating faithful, executable SimPy-based queueing simulations with LLMs by presenting a three-stage fine-tuning framework that adapts open-source models (Qwen-Coder-7B and DeepSeek-Coder-6.7B) to queueing-domain tasks. A Category-Template Framework (CTF) drives Stage I data construction and initial supervised fine-tuning, Stage II adds masked-completion training to deepen mechanism understanding, and Stage III applies Direct Preference Optimization to align generation with correct queueing logic. On a 600-task held-out benchmark, the approach yields substantial gains in executability, output-format compliance, and instruction-consistency (e.g., Qwen: executability 80.4%→86% and consistency 0%→76.8%; DeepSeek: executability 26.2%→75% and consistency 0%→62.3%), demonstrating that domain-focused fine-tuning makes compact open models reliable SimPy simulators. The results support the practical use of open-source LLMs for education, research, and decision-support in queueing-context modeling while preserving cost and privacy advantages over closed-source alternatives.

Abstract

The Python package SimPy is widely used for modeling queueing systems due to its flexibility, simplicity, and smooth integration with modern data analysis and optimization frameworks. Recent advances in large language models (LLMs) have shown strong ability in generating clear and executable code, making them powerful and suitable tools for writing SimPy queueing simulation code. However, directly employing closed-source models like GPT-4o to generate such code may lead to high computational costs and raise data privacy concerns. To address this, we fine-tune two open-source LLMs, Qwen-Coder-7B and DeepSeek-Coder-6.7B, on curated SimPy queueing data, which enhances their code-generating performance in executability, output-format compliance, and instruction-code consistency. Particularly, we proposed a multi-stage fine-tuning framework comprising two stages of supervised fine-tuning (SFT) and one stage of direct preference optimization (DPO), progressively enhancing the model's ability in SimPy-based queueing simulation code generation. Extensive evaluations demonstrate that both fine-tuned models achieve substantial improvements in executability, output-format compliance, and instruct consistency. These results confirm that domain-specific fine-tuning can effectively transform compact open-source code models into reliable SimPy simulation generators which provide a practical alternative to closed-source LLMs for education, research, and operational decision support.

Figures (10)

• Figure 1: Three instruction templates for the batch arrivals category.
• Figure 2: Representative code fragments from the batch arrivals category illustrating Three Implementation Templates: procedural (T0), object-oriented (T1), and functional (T2).
• Figure 3: Stage I data construction pipeline of the Category-Template Framework (CTF).
• Figure 4: Representative masked-completion tasks in Stage II for the batch arrivals category.
• Figure 5: Stage II masked-completion data construction pipeline.