Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Reinforcement Learning and Optimal Transport for the Traveling Salesman Problem

Yong Liang Goh, Wee Sun Lee, Xavier Bresson, Thomas Laurent, Nicholas Lim

TL;DR

The paper addresses the challenge of solving large TSP instances by blending reinforcement learning with edge prediction in a Transformer-based encoder. By removing a learnable decoder and introducing a differentiable Sinkhorn-based optimal transport layer, the model outputs valid edge probabilities quickly while enforcing assignment constraints during training. The Sinkhorn bias yields notable gains in optimality gaps (up to a 3x improvement on TSP50) with minimal impact on inference time, and the approach scales more favorably for larger instances. This work demonstrates the practical benefits of integrating differentiable optimal transport into deep RL for combinatorial problems and highlights avenues for integrating search strategies during training and inference to scale further.

Abstract

The traveling salesman problem is a fundamental combinatorial optimization problem with strong exact algorithms. However, as problems scale up, these exact algorithms fail to provide a solution in a reasonable time. To resolve this, current works look at utilizing deep learning to construct reasonable solutions. Such efforts have been very successful, but tend to be slow and compute intensive. This paper exemplifies the integration of entropic regularized optimal transport techniques as a layer in a deep reinforcement learning network. We show that we can construct a model capable of learning without supervision and inferences significantly faster than current autoregressive approaches. We also empirically evaluate the benefits of including optimal transport algorithms within deep learning models to enforce assignment constraints during end-to-end training.

Combining Reinforcement Learning and Optimal Transport for the Traveling Salesman Problem

TL;DR

The paper addresses the challenge of solving large TSP instances by blending reinforcement learning with edge prediction in a Transformer-based encoder. By removing a learnable decoder and introducing a differentiable Sinkhorn-based optimal transport layer, the model outputs valid edge probabilities quickly while enforcing assignment constraints during training. The Sinkhorn bias yields notable gains in optimality gaps (up to a 3x improvement on TSP50) with minimal impact on inference time, and the approach scales more favorably for larger instances. This work demonstrates the practical benefits of integrating differentiable optimal transport into deep RL for combinatorial problems and highlights avenues for integrating search strategies during training and inference to scale further.

Abstract

The traveling salesman problem is a fundamental combinatorial optimization problem with strong exact algorithms. However, as problems scale up, these exact algorithms fail to provide a solution in a reasonable time. To resolve this, current works look at utilizing deep learning to construct reasonable solutions. Such efforts have been very successful, but tend to be slow and compute intensive. This paper exemplifies the integration of entropic regularized optimal transport techniques as a layer in a deep reinforcement learning network. We show that we can construct a model capable of learning without supervision and inferences significantly faster than current autoregressive approaches. We also empirically evaluate the benefits of including optimal transport algorithms within deep learning models to enforce assignment constraints during end-to-end training.
Paper Structure (16 sections, 11 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 16 sections, 11 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Autoregressive Approaches
  4. Nonautoregessive Approaches
  5. Combining RL and Edge Prediction
  6. The Transformer Encoder
  7. The Softmax Decoder
  8. The Sinkhorn Decoder
  9. Decoding Process
  10. Learning Algorithm
  11. Inference
  12. Experiments & Results
  13. Experiment Setup
  14. Results
  15. Discussion & Future Work
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of our approach. We use a standard multi-headed transformer architecture as the encoder. An $n \times n$ heatmap is then produced through vector outer products, which is converted into a valid distribution by cuturi2013sinkhorn's algorithm. A valid tour is constructed by decoding. All operations are differentiable and the model is trained end-to-end.
  • Figure 2: cuturi2013sinkhorn's algorithm applied to an output from our model for a TSP50 problem. Best viewed in color. Top left depicts the original probabilities, while the respective values of $\lambda$ are indicated accordingly. Colors range from red (close to 0) to blue (close to 1), and diagonals have been masked away as it is not a valid choice. Without any enforcement of assignment constraints, the network's predictions become concentrated, and following these probabilities does not result in a valid tour. Enforcing these constraints results in the distribution spreading out across the cities. Increasing $\lambda$ results in less entropic regularization and the optimal transport map becomes closer to the assignment solution, with strong probability at a single point and others coming close to 0. With more regularization, the problem moves away from the corner of the simplex and the distribution becomes more even and uniform.