Heuristics for Combinatorial Optimization via Value-based Reinforcement Learning: A Unified Framework and Analysis
Orit Davidovich, Shimrit Shtern, Segev Wasserkrug, Nimrod Megiddo
TL;DR
<3-5 sentence high-level summary> The paper addresses the theoretical gap for using value-based reinforcement learning to solve combinatorial optimization by proposing a unified translation of CO problems into undiscounted Markov decision processes via Karp-Held theory. It develops convergence and error-analysis results for projected and fitted value iteration within this CO-MDP framework, enabling explicit CO-optimality guarantees from ε-approximations of V^*. The work applies the framework to canonical problems like the Traveling Salesman Problem and the Shortest Path Problem and discusses sample-average approximation and estimation procedures, providing guidance on state embeddings and algorithm design. This delivers a rigorous foundation for RL-based CO solvers and informs practical aspects such as approximation schemes, contraction conditions, and problem-specific state-space constructions.
Abstract
Since the 1990s, considerable empirical work has been carried out to train statistical models, such as neural networks (NNs), as learned heuristics for combinatorial optimization (CO) problems. When successful, such an approach eliminates the need for experts to design heuristics per problem type. Due to their structure, many hard CO problems are amenable to treatment through reinforcement learning (RL). Indeed, we find a wealth of literature training NNs using value-based, policy gradient, or actor-critic approaches, with promising results, both in terms of empirical optimality gaps and inference runtimes. Nevertheless, there has been a paucity of theoretical work undergirding the use of RL for CO problems. To this end, we introduce a unified framework to model CO problems through Markov decision processes (MDPs) and solve them using RL techniques. We provide easy-to-test assumptions under which CO problems can be formulated as equivalent undiscounted MDPs that provide optimal solutions to the original CO problems. Moreover, we establish conditions under which value-based RL techniques converge to approximate solutions of the CO problem with a guarantee on the associated optimality gap. Our convergence analysis provides: (1) a sufficient rate of increase in batch size and projected gradient descent steps at each RL iteration; (2) the resulting optimality gap in terms of problem parameters and targeted RL accuracy; and (3) the importance of a choice of state-space embedding. Together, our analysis illuminates the success (and limitations) of the celebrated deep Q-learning algorithm in this problem context.