Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvement

TL;DR

This work introduces a method that combines round-wise Stochastic Beam Search with an update strategy derived from a provable policy improvement that refines the policy between rounds by utilizing the advantage of the sampled sequences with almost no computational overhead.

Abstract

Current methods for end-to-end constructive neural combinatorial optimization usually train a policy using behavior cloning from expert solutions or policy gradient methods from reinforcement learning. While behavior cloning is straightforward, it requires expensive expert solutions, and policy gradient methods are often computationally demanding and complex to fine-tune. In this work, we bridge the two and simplify the training process by sampling multiple solutions for random instances using the current model in each epoch and then selecting the best solution as an expert trajectory for supervised imitation learning. To achieve progressively improving solutions with minimal sampling, we introduce a method that combines round-wise Stochastic Beam Search with an update strategy derived from a provable policy improvement. This strategy refines the policy between rounds by utilizing the advantage of the sampled sequences with almost no computational overhead. We evaluate our approach on the Traveling Salesman Problem and the Capacitated Vehicle Routing Problem. The models trained with our method achieve comparable performance and generalization to those trained with expert data. Additionally, we apply our method to the Job Shop Scheduling Problem using a transformer-based architecture and outperform existing state-of-the-art methods by a wide margin.

Figures (6)

• Figure 1: Example of incremental sampling using a trie. Nodes represent partial sequences (here, letters) and contain their total probabilities. i.) The sequence 'BDG' is sampled from the model, and we subtract its total probability from its ancestors. ii.) Updated trie after erasing 'BDG'.
• Figure 2: Example of update (\ref{['eq:full_gd_policy_update']}) with beam width $k = 3$ and $\sigma = 1$. Nodes represent partial sequences and their total probabilities; the numbers $\{1, \dots, 9\}$ below the leaf nodes are their corresponding objective function evaluation. i.) We sample $k$ leaf nodes with SBS and obtain the objective function estimate $\mu$ with (\ref{['eq:normalized_threshold_estimator']}). ii.) We mark each leaf as sampled by removing its total probability from its ancestors. At the same time (equivalent to adding to the log-probability), we multiply the ancestors' reduced probability by the exponential of the leaf's advantage. iii.) Trie after normalization, corresponding to $\pi'$.
• Figure 3: Sampling results for different training checkpoints, with the corresponding greedy performance of the model. For Sample WR, we sample $32\cdot n$ sequences with replacement, where $n$ is the number of rounds. For TSP and CVRP, each data point corresponds to the average best solution across 100 instances. For JSSP, we evaluate the Taillard instances of the corresponding size. Sampling for each data point is repeated 20 times; shades denote standard errors.
• Figure 4: Comparison of the practical policy update (\ref{['eq:full_gd_policy_update']}) 'GD' with the theoretical update (\ref{['eq:theory_policy_update']}) 'Theory GD' across various model checkpoints and beam widths $k \in \{16,32,64\}$. For TSP and CVRP, each data point corresponds to the average best solution across 100 instances. For JSSP, we evaluate the Taillard instances of the corresponding size. Nucleus sampling is switched off. Sampling for each data point is repeated 10 times; shades denote standard errors.
• Figure 5: Example of a problem instance with $J=2$ and $M=4$. The operations' processing times and machines are shown in the table on the left. The first two operations have already been scheduled with the partial schedule $(1, 1, 2, 2)$. The corresponding Gantt chart and availability times are shown on the right.