Solver-in-the-Loop: MDP-Based Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

TL;DR

This paper addresses the gap in Operations Research benchmarks by introducing solver-in-the-loop evaluation where LLMs interact with a deterministic IIS oracle to diagnose and repair infeasible models. It proposes two complementary benchmarks, OR-Debug-Bench for iterative self-correction and OR-Bias-Bench for behavioral rationality under ID/OOD distribution shifts, and couples them with RLVR-style domain-specific training (GRPO with a Process Reward Model and curriculum learning). The results show that domain-specific 8B models can surpass frontier APIs in recovery rate and diagnostic accuracy while achieving greater efficiency, and curriculum learning yields notable bias mitigation in OOD settings. Overall, the work demonstrates that process-level evaluation with verifiable solvers enables scalable, targeted training that improves reliability and generalization in OR tasks.

Abstract

Operations Research practitioners routinely debug infeasible models through an iterative process: analyzing Irreducible Infeasible Subsystems (\IIS{}), identifying constraint conflicts, and systematically repairing formulations until feasibility is achieved. Yet existing LLM benchmarks evaluate OR as one-shot translation -- given a problem description, generate solver code -- ignoring this diagnostic loop entirely. We introduce two benchmarks that place the \textbf{solver in the evaluation loop}. \textbf{\ORDebug{}} evaluates iterative self-correction through 5,000+ problems spanning 9 error types; each repair action triggers solver re-execution and \IIS{} recomputation, providing deterministic, verifiable feedback. \textbf{\ORBias{}} evaluates behavioral rationality through 2,000 newsvendor instances (1,000 ID + 1,000 OOD), measuring systematic deviations from closed-form optimal policies. Across 26 models and 12,000+ samples, we find that domain-specific RLVR training enables an 8B model to surpass frontier APIs: 95.3\% vs 86.2\% recovery rate (+9.1\%), 62.4\% vs 47.8\% diagnostic accuracy (+14.6\%), and 2.25 vs 3.78 steps to resolution (1.7$\times$ faster). On \ORBias{}, curriculum training achieves the only negative ID$\rightarrow$OOD bias drift among models evaluated (-9.6\%), reducing systematic bias by 48\% (from 20.0\% to 10.4\%). These results demonstrate that process-level evaluation with verifiable oracles enables targeted training that outperforms scale.

Figures (5)

• Figure 1: Evaluation paradigms compared. Top: Static translation benchmarks evaluate one-shot code generation with no execution feedback. Bottom: Our solver-in-the-loop approach enables iterative self-correction through IIS feedback.
• Figure 2: Two-phase benchmark framework. Phase I (OR-Debug-Bench): Iterative debugging where the agent receives Gurobi IIS feedback and repairs infeasible code. Phase II (OR-Bias-Bench): Inventory decision-making verified against the closed-form optimal policy $Q^* = F^{-1}(\text{CR})$.
• Figure 3: Example OR-Debug-Bench episode. Left: The agent receives a sabotaged LP where minimum requirements ($60+50+0=110$) exceed capacity ($100$). Right: The agent (1) attempts optimization, (2) computes IIS, (3) reasons about the conflict, (4) relaxes the key constraint, and (5) achieves Optimal status in 2 repair steps.
• Figure 4: Training pipeline overview. Track 1 trains the OR-Debug-Bench model: SFT on teacher trajectories followed by GRPO with composite reward and optional PRM supervision. Track 2 trains the OR-Bias-Bench model: SFT on rational responses followed by a three-stage curriculum (Extreme $\rightarrow$ Boundary $\rightarrow$ Full) that targets pull-to-center bias.
• Figure 5: Recovery Rate vs. attempt budget $k$. Qwen3-8B-GRPO achieves at $k=5$ (95.3%) what the best API model requires $k>10$ to approach.