Preference Optimization for Combinatorial Optimization Problems
Mingjun Pan, Guanquan Lin, You-Wei Luo, Bin Zhu, Zhien Dai, Lijun Sun, Chun Yuan
TL;DR
The paper tackles the difficulty of applying reinforcement learning to combinatorial optimization problems (COPs) due to diminishing reward differences and enormous action spaces. It introduces Preference Optimization (PO), a framework that converts quantitative rewards into qualitative preferences using an entropy-regularized objective and a reparameterized latent reward, with trajectory preferences modeled via Bradley-Terry, Thurstone, or Plackett-Luce constructions. A core contribution is deriving an update mechanism that aligns policy learning with relative preferences while avoiding full enumeration of actions, plus integrating local search into fine-tuning to escape local optima without adding inference time. Empirically, PO yields faster convergence and higher-quality solutions across TSP, CVRP, and FFSP, and demonstrates strong zero-shot generalization and compatibility with LS-based refinements. This approach offers a scalable, reward-scale-invariant path to robust neural solvers for COPs and suggests avenues for extending PO to multi-objective optimization problems.
Abstract
Reinforcement Learning (RL) has emerged as a powerful tool for neural combinatorial optimization, enabling models to learn heuristics that solve complex problems without requiring expert knowledge. Despite significant progress, existing RL approaches face challenges such as diminishing reward signals and inefficient exploration in vast combinatorial action spaces, leading to inefficiency. In this paper, we propose Preference Optimization, a novel method that transforms quantitative reward signals into qualitative preference signals via statistical comparison modeling, emphasizing the superiority among sampled solutions. Methodologically, by reparameterizing the reward function in terms of policy and utilizing preference models, we formulate an entropy-regularized RL objective that aligns the policy directly with preferences while avoiding intractable computations. Furthermore, we integrate local search techniques into the fine-tuning rather than post-processing to generate high-quality preference pairs, helping the policy escape local optima. Empirical results on various benchmarks, such as the Traveling Salesman Problem (TSP), the Capacitated Vehicle Routing Problem (CVRP) and the Flexible Flow Shop Problem (FFSP), demonstrate that our method significantly outperforms existing RL algorithms, achieving superior convergence efficiency and solution quality.