Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProOPF: Benchmarking and Improving LLMs for Professional-Grade Power Systems Optimization Modeling

Chao Shen, Zihan Guo, Xu Wan, Zhenghao Yang, Yifan Zhang, Wengi Huang, Jie Song, Zongyan Zhang, Mingyang Sun

TL;DR

ProOPF introduces ProOPF-D and ProOPF-B to benchmark how well Large Language Models can handle professional-grade power system optimization modeling, specifically Optimal Power Flow (OPF). It reframes OPF as a modification-based problem, using a canonical base model and a structured set of parameter patches and structural extensions, enabling scalable, physics-consistent data generation across four difficulty levels. Empirically, state-of-the-art LLMs excel at explicit parameter changes but struggle with semantic parameter inference and complex structural modifications; fine-tuning on ProOPF-D substantially improves performance on harder levels, though abstract modeling remains challenging. The work provides a foundation for domain-specific data generation and evaluation, guiding future research toward physics-informed supervision and more robust, industry-ready LLM capabilities for power-system optimization tasks.

Abstract

Growing renewable penetration introduces substantial uncertainty into power system operations, necessitating frequent adaptation of dispatch objectives and constraints and challenging expertise-intensive, near-real-time modeling workflows. Large Language Models (LLMs) provide a promising avenue for automating this process by translating natural-language (NL) operational requirements into executable optimization models via semantic reasoning and code synthesis. Yet existing LLM datasets and benchmarks for optimization modeling primarily target coarse-grained cross-domain generalization, offering limited, rigorous evaluation in power-system settings, particularly for Optimal Power Flow (OPF). We therefore introduce \textbf{ProOPF-D} and \textbf{ProOPF-B}, a dataset and benchmark for professional-grade OPF modeling: ProOPF-D contains 12K instances pairing NL requests with parameter adjustments and structural extensions to a canonical OPF, together with executable implementations; ProOPF-B provides 121 expert-annotated test cases with ground-truth code, enabling end-to-end evaluation under both concrete and abstract OPF modeling regimes.

ProOPF: Benchmarking and Improving LLMs for Professional-Grade Power Systems Optimization Modeling

TL;DR

ProOPF introduces ProOPF-D and ProOPF-B to benchmark how well Large Language Models can handle professional-grade power system optimization modeling, specifically Optimal Power Flow (OPF). It reframes OPF as a modification-based problem, using a canonical base model and a structured set of parameter patches and structural extensions, enabling scalable, physics-consistent data generation across four difficulty levels. Empirically, state-of-the-art LLMs excel at explicit parameter changes but struggle with semantic parameter inference and complex structural modifications; fine-tuning on ProOPF-D substantially improves performance on harder levels, though abstract modeling remains challenging. The work provides a foundation for domain-specific data generation and evaluation, guiding future research toward physics-informed supervision and more robust, industry-ready LLM capabilities for power-system optimization tasks.

Abstract

Growing renewable penetration introduces substantial uncertainty into power system operations, necessitating frequent adaptation of dispatch objectives and constraints and challenging expertise-intensive, near-real-time modeling workflows. Large Language Models (LLMs) provide a promising avenue for automating this process by translating natural-language (NL) operational requirements into executable optimization models via semantic reasoning and code synthesis. Yet existing LLM datasets and benchmarks for optimization modeling primarily target coarse-grained cross-domain generalization, offering limited, rigorous evaluation in power-system settings, particularly for Optimal Power Flow (OPF). We therefore introduce \textbf{ProOPF-D} and \textbf{ProOPF-B}, a dataset and benchmark for professional-grade OPF modeling: ProOPF-D contains 12K instances pairing NL requests with parameter adjustments and structural extensions to a canonical OPF, together with executable implementations; ProOPF-B provides 121 expert-annotated test cases with ground-truth code, enabling end-to-end evaluation under both concrete and abstract OPF modeling regimes.
Paper Structure (49 sections, 27 equations, 5 figures, 3 tables, 7 algorithms)

This paper contains 49 sections, 27 equations, 5 figures, 3 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Power System Optimization Modeling
  4. Foundation: Optimal Power Flow Problem
  5. Base Model Modification
  6. ProOPF-D: Expert-Curated Dataset via Modification-Based OPF Representation
  7. Structured Sample Representation
  8. Multi-Level Difficulty Taxonomy and Synthesis
  9. ProOPF-B: Benchmark for LLM Performance on OPF Modeling Tasks
  10. Experiment and Analysis
  11. Baselines and Settings
  12. Baslines Evaluation on ProOPF-B
  13. Supervised Fine-tuning on ProOPF-D
  14. Conclusion and Future Work
  15. Details of Model Specification in ProOPF-D
  16. ...and 34 more sections

Figures (5)

  • Figure 1: From cross-domain to within-domain generalization in LLM-based optimization. Existing works emphasize coarse-grained generalization across heterogeneous optimization tasks, whereas ProOPF-B/D targets fine-grained, within-domain generalization in OPF through parametric and structural formulation modifications.
  • Figure 2: The ProOPF-D dataset construction pipeline. $\mathcal{L}_1$ generates samples by directly instantiating OPF models from explicitly specified parameter patches. $\mathcal{L}_2$ synthesizes scenario-driven samples by mapping qualitative operational descriptions to parameter modification directions using expert-curated scenario trees. $\mathcal{L}_3$ extends the base OPF formulation with expert-designed structural variants combined with explicit parameter updates. $\mathcal{L}_4$ integrates semantic parameter inference and structural extensions, representing the most challenging expert-level modeling setting. Finally, the synthetic data undergoes Data Cleaning and Text Refinement, resulting in a total dataset of 12k samples evenly distributed across four levels (3k each).
  • Figure 3: Expert-curated scenario trees for $\mathcal{L}_2$ synthesis. Top: Hierarchical structure from event-level (E-level) through mechanism-level (M-level) nodes to leaf nodes encoding parameter types and modification trends. Bottom: Retrieval process where $\operatorname{Retrieve}(\mathcal{T} | \operatorname{par}(\delta_k), \operatorname{dir}(\delta_k))$ matches parameter patch $\delta_k$ to leaf nodes, and the root-to-leaf path forms scenario fragment $c_k$.
  • Figure 4: Six-Dimensional Capability Radar Chart. Each axis represents a fundamental competency in OPF modeling (see \ref{['app:six_dimensions']} for detailed definitions). All values are expressed as percentages.
  • Figure 5: Schematic comparison of evaluation workflows for concrete (Levels 1/3) and abstract (Levels 2/4) OPF modeling in ProOPF-B, illustrating the key difference in validation procedures.