Execution-Verified Reinforcement Learning for Optimization Modeling

Abstract

Automating optimization modeling with LLMs is a promising path toward scalable decision intelligence, but existing approaches either rely on agentic pipelines built on closed-source LLMs with high inference latency, or fine-tune smaller LLMs using costly process supervision that often overfits to a single solver API. Inspired by reinforcement learning with verifiable rewards, we propose Execution-Verified Optimization Modeling (EVOM), an execution-verified learning framework that treats a mathematical programming solver as a deterministic, interactive verifier. Given a natural-language problem and a target solver, EVOM generates solver-specific code, executes it in a sandboxed harness, and converts execution outcomes into scalar rewards, optimized with GRPO and DAPO in a closed-loop generate-execute-feedback-update process. This outcome-only formulation removes the need for process-level supervision, and enables cross-solver generalization by switching the verification environment rather than reconstructing solver-specific datasets. Experiments on NL4OPT, MAMO, IndustryOR, and OptiBench across Gurobi, OR-Tools, and COPT show that EVOM matches or outperforms process-supervised SFT, supports zero-shot solver transfer, and achieves effective low-cost solver adaptation by continuing training under the target solver backend.

Figures (8)

• Figure 1: Comparison of Execution-Verified Optimization Modeling with previous methods.
• Figure 2: Impact of explicit reasoning on EVOM performance across different benchmarks.
• Figure 3: GRPO vs. DAPO under identical training setup. Accuracy curves across benchmarks are nearly identical, indicating low sensitivity to the optimizer choice in execution-verified training.
• Figure 4: Training dynamics of execution-verified learning.
• Figure 5: Solver-conditioned prompt template with a fixed two-block output format for parsing and execution.