OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling
Yitian Chen, Cheng Cheng, Yinan Sun, Zi Ling, Dongdong Ge
TL;DR
The paper tackles the challenge of rigorously evaluating LLMs in optimization modeling by introducing OPT-Engine, a solver-verified, extensible benchmark framework that generates scalable, verifiable optimization instances across $10$ OR tasks (LP and MIP). It systematically compares tool-integrated reasoning (TIR) with pure-text reasoning (PTR) using frontier models and open-source variants, demonstrating that tool integration sustains accuracy as complexity grows while pure-text reasoning hits a performance ceiling. A key finding is that the primary bottleneck lies in auto-formulation and constraint grounding—especially when augmenting constraints—rather than in problem comprehension or objective perturbations, highlighting the need for stronger grounding and solver-integration in next-generation LLMs. The framework’s scalability, rigorous diagnostic tests (linguistic variation, objective shifts, and constraint augmentation), and solver-verified pipeline provide a practical path toward industrial-scale optimization with LLMs and offer actionable guidance for developing more robust, autonomous optimization agents.
Abstract
Large Language Models (LLMs) have demonstrated impressive progress in optimization modeling, fostering a rapid expansion of new methodologies and evaluation benchmarks. However, the boundaries of their capabilities in automated formulation and problem solving remain poorly understood, particularly when extending to complex, real-world tasks. To bridge this gap, we propose OPT-ENGINE, an extensible benchmark framework designed to evaluate LLMs on optimization modeling with controllable and scalable difficulty levels. OPT-ENGINE spans 10 canonical tasks across operations research, with five Linear Programming and five Mixed-Integer Programming. Utilizing OPT-ENGINE, we conduct an extensive study of LLMs' reasoning capabilities, addressing two critical questions: 1.) Do LLMs' performance remain robust when generalizing to out-of-distribution optimization tasks that scale in complexity beyond current benchmark levels? and 2.) At what stage, from problem interpretation to solution generation, do current LLMs encounter the most significant bottlenecks? Our empirical results yield two key insights: first, tool-integrated reasoning with external solvers exhibits significantly higher robustness as task complexity escalates, while pure-text reasoning reaches a ceiling; second, the automated formulation of constraints constitutes the primary performance bottleneck. These findings provide actionable guidance for developing next-generation LLMs for advanced optimization. Our code is publicly available at \textcolor{blue}{https://github.com/Cardinal-Operations/OPTEngine}.