StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models

TL;DR

This paper tackles the challenge of solving complex Operations Research problems with LLMs by addressing credit assignment in outcome-based rewards and the myopic nature of discriminative process supervision. It introduces StepORLM, a self-evolving framework where a policy model and a generative process reward model (GenPRM) co-evolve, using dual feedback from an external solver and holistic process critique to train via Weighted Direct Preference Optimization and GenPRM refinement. The approach achieves state-of-the-art performance with an 8B-parameter model across six OR benchmarks and demonstrates that the co-evolved GenPRM can act as a universal inference-time verifier, boosting the performance of other models as well. These results highlight a scalable path toward reliable, process-sound OR reasoning in LLMs, with practical implications for deploying specialized, verifier-augmented LLMs in real-world optimization tasks.

Abstract

Large Language Models (LLMs) have shown promising capabilities for solving Operations Research (OR) problems. While reinforcement learning serves as a powerful paradigm for LLM training on OR problems, existing works generally face two key limitations. First, outcome reward suffers from the credit assignment problem, where correct final answers can reinforce flawed reasoning. Second, conventional discriminative process supervision is myopic, failing to evaluate the interdependent steps of OR modeling holistically. To this end, we introduce StepORLM, a novel self-evolving framework with generative process supervision. At its core, StepORLM features a co-evolutionary loop where a policy model and a generative process reward model (GenPRM) iteratively improve on each other. This loop is driven by a dual-feedback mechanism: definitive, outcome-based verification from an external solver, and nuanced, holistic process evaluation from the GenPRM. The combined signal is used to align the policy via Weighted Direct Preference Optimization (W-DPO) and simultaneously refine the GenPRM. Our resulting 8B-parameter StepORLM establishes a new state-of-the-art across six benchmarks, significantly outperforming vastly larger generalist models, agentic methods, and specialized baselines. Moreover, the co-evolved GenPRM is able to act as a powerful and universally applicable process verifier, substantially boosting the inference scaling performance of both our own model and other existing LLMs.

Figures (5)

• Figure 1: The illustration of (a) the credit assignment problem of outcome reward and (b) the myopic issue of discriminative process supervision, which motivates the application of generative process supervision for operations research language models.
• Figure 2: The illustration of data synthesis pipeline for the warm-up stage. A teacher LLM generates diverse OR problems and their step-by-step reasoning solutions. Each solution is rigorously validated and filtered by an external solver, with an automated refinement loop to ensure the quality. This verified corpus is then used to train the initial policy model via SFT.
• Figure 3: The co-evolutionary loop of StepORLM. At each iteration, the policy model $\pi_{\theta}$ generates multiple trajectories. The feedback from both the external solver (outcome) and the GenPRM $\rho_{\theta}$ (process) is used to create training data that simultaneously refines the policy via W-DPO and improves the GenPRM via SFT, fostering reciprocal improvement.
• Figure 4: The analysis on the self-evolving process by tracking the performance (Pass@1 accuracy) at each training iteration. The relative improvement of current iteration over the previous one is demonstrated on the corresponding bar.
• Figure 5: A simplified illustration of problem 74 in ComplexLP