Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Decision Transformers for Neural Combinatorial Optimization: Surpassing Heuristics on the Traveling Salesman Problem

Hironori Ohigashi, Shinichiro Hamada

Abstract

Combinatorial optimization problems like the Traveling Salesman Problem are critical in industry yet NP-hard. Neural Combinatorial Optimization has shown promise, but its reliance on online reinforcement learning (RL) hampers deployment and underutilizes decades of algorithmic knowledge. We address these limitations by applying the offline RL framework, Decision Transformer, to learn superior strategies directly from datasets of heuristic solutions; it aims to not only to imitate but to synthesize and outperform them. Concretely, we (i) integrate a Pointer Network to handle the instance-dependent, variable action space of node selection, and (ii) employ expectile regression for optimistic conditioning of Return-to-Go, which is crucial for instances with widely varying optimal values. Experiments show that our method consistently produces higher-quality tours than the four classical heuristics it is trained on, demonstrating the potential of offline RL to unlock and exceed the performance embedded in existing domain knowledge.

Offline Decision Transformers for Neural Combinatorial Optimization: Surpassing Heuristics on the Traveling Salesman Problem

Abstract

Combinatorial optimization problems like the Traveling Salesman Problem are critical in industry yet NP-hard. Neural Combinatorial Optimization has shown promise, but its reliance on online reinforcement learning (RL) hampers deployment and underutilizes decades of algorithmic knowledge. We address these limitations by applying the offline RL framework, Decision Transformer, to learn superior strategies directly from datasets of heuristic solutions; it aims to not only to imitate but to synthesize and outperform them. Concretely, we (i) integrate a Pointer Network to handle the instance-dependent, variable action space of node selection, and (ii) employ expectile regression for optimistic conditioning of Return-to-Go, which is crucial for instances with widely varying optimal values. Experiments show that our method consistently produces higher-quality tours than the four classical heuristics it is trained on, demonstrating the potential of offline RL to unlock and exceed the performance embedded in existing domain knowledge.

Paper Structure

This paper contains 19 sections, 4 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. TSP formulation
  4. Application of DT
  5. Action representation via Pointer Network
  6. RTG prediction
  7. Learning objective and loss function
  8. Experimental results
  9. Experimental setup
  10. Prediction performance of the proposed method
  11. Performance comparison with behavior cloning
  12. Effect of expectile regression on RTG
  13. Exploring optimal RTG
  14. Discussion and Conclusion
  15. Experimental details
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Overview of the proposed method's architecture. The transformer encoder calculates a node embedding vector from the coordinates of the graph nodes. As observation $o_t$ and action $a_t$, the embedding vectors $\boldsymbol{f}^{e}_{t}$ and $\boldsymbol{f}^{e}_{t+1}$ of the nodes transitioned at time $t$ and $t+1$, respectively, are input to the causal transformer decoder along with the RTG $\hat{R}_t$. The value $\tilde{R}_t$ is the RTG prediction by the causal transformer decoder.