Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning Methods for Neighborhood Selection in Local Search

Yannick Molinghen, Augustin Delecluse, Renaud De Landtsheer, Stefano Michelini

TL;DR

This work investigates reinforcement learning approaches for selecting local-search neighborhoods in NP-hard combinatorial optimization across TSP, PDPTW, and CSP. It systematically compares multi-armed bandits (ε-greedy, UCB) and deep RL methods (DDQN, PPO) using three reward schemes designed to align learning with the LS objective, including a theoretically ideal reward, a log-adjusted reward, and a duration-aware reward. Results show substantial performance variation by problem and reward design, with ε-greedy providing robust, efficient performance in many cases, while DRL methods offer potential gains only with longer runtimes and carefully crafted rewards. The findings stress practical constraints: DRL is not a plug-in replacement for classical LS heuristics and should be weighed against strong, low-overhead baselines, particularly under realistic time budgets. Future work is encouraged to explore generalization across solvers, pretraining trade-offs, and broader integration strategies that preserve simplicity and scalability.

Abstract

Reinforcement learning has recently gained traction as a means to improve combinatorial optimization methods, yet its effectiveness within local search metaheuristics specifically remains comparatively underexamined. In this study, we evaluate a range of reinforcement learning-based neighborhood selection strategies -- multi-armed bandits (upper confidence bound, $ε$-greedy) and deep reinforcement learning methods (proximal policy optimization, double deep $Q$-network) -- and compare them against multiple baselines across three different problems: the traveling salesman problem, the pickup and delivery problem with time windows, and the car sequencing problem. We show how search-specific characteristics, particularly large variations in cost due to constraint violation penalties, necessitate carefully designed reward functions to provide stable and informative learning signals. Our extensive experiments reveal that algorithm performance varies substantially across problems, although that $ε$-greedy consistently ranks among the best performers. In contrast, the computational overhead of deep reinforcement learning approaches only makes them competitive with a substantially longer runtime. These findings highlight both the promise and the practical limitations of deep reinforcement learning in local search.

Reinforcement Learning Methods for Neighborhood Selection in Local Search

TL;DR

This work investigates reinforcement learning approaches for selecting local-search neighborhoods in NP-hard combinatorial optimization across TSP, PDPTW, and CSP. It systematically compares multi-armed bandits (ε-greedy, UCB) and deep RL methods (DDQN, PPO) using three reward schemes designed to align learning with the LS objective, including a theoretically ideal reward, a log-adjusted reward, and a duration-aware reward. Results show substantial performance variation by problem and reward design, with ε-greedy providing robust, efficient performance in many cases, while DRL methods offer potential gains only with longer runtimes and carefully crafted rewards. The findings stress practical constraints: DRL is not a plug-in replacement for classical LS heuristics and should be weighed against strong, low-overhead baselines, particularly under realistic time budgets. Future work is encouraged to explore generalization across solvers, pretraining trade-offs, and broader integration strategies that preserve simplicity and scalability.

Abstract

Reinforcement learning has recently gained traction as a means to improve combinatorial optimization methods, yet its effectiveness within local search metaheuristics specifically remains comparatively underexamined. In this study, we evaluate a range of reinforcement learning-based neighborhood selection strategies -- multi-armed bandits (upper confidence bound, -greedy) and deep reinforcement learning methods (proximal policy optimization, double deep -network) -- and compare them against multiple baselines across three different problems: the traveling salesman problem, the pickup and delivery problem with time windows, and the car sequencing problem. We show how search-specific characteristics, particularly large variations in cost due to constraint violation penalties, necessitate carefully designed reward functions to provide stable and informative learning signals. Our extensive experiments reveal that algorithm performance varies substantially across problems, although that -greedy consistently ranks among the best performers. In contrast, the computational overhead of deep reinforcement learning approaches only makes them competitive with a substantially longer runtime. These findings highlight both the promise and the practical limitations of deep reinforcement learning in local search.
Paper Structure (39 sections, 24 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 39 sections, 24 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. Local Search
  4. Move operators and neighborhoods:
  5. Reinforcement Learning
  6. Bandit setting (single-state):
  7. Markov decision process (multi-state):
  8. Related work
  9. METHODOLOGY
  10. Acceptance criterion:
  11. Constraints:
  12. Tabu list:
  13. Restarts:
  14. Reinforcement Learning Formulation
  15. Bandit Algorithms
  16. ...and 24 more sections

Figures (3)

  • Figure 1: First stages of the search for the random and RR move selectors on a small TSP instance (24 cities). The top plot shows the value of $f$ over the course of the search, and the bottom plot shows which move operator has been tried but not selected because they did not improve the solution (grayed), and the ones that have been selected (colored) at each iteration.
  • Figure 2: BSF, $\epsilon$-greedy and UCB move operator selection on the TSP. The weights are normalized between [0, 1].
  • Figure 3: Primal gap (lower is better) over time averaged across all instances and 10 random seeds. Colors account for the move selection algorithm and the line styles for the reward function. The time budget is 15 minutes for the PDPTW and the TSP, and 90 minutes for the CSP. The CSP is the only problem where DRL agents perform the best.