Probing Neural Combinatorial Optimization Models

TL;DR

The paper tackles the interpretability of neural combinatorial optimization (NCO) models by designing problem-specific probing tasks and introducing CS-Probing to quantify both the magnitude and significance of informative embedding dimensions. Experiments on AM, POMO, and LEHD across TSP and CVRP show that NCO embeddings encode both low-level spatial relationships and high-level decision strategies, with LEHD exhibiting strong, sparse dimensional usage and distinct inductive biases. CS-Probing provides direct evidence of generalization mechanisms, revealing that models with robust performance tend to reuse the same embedding dimensions across tasks and scales, while disorganized knowledge across dimensions correlates with degradation. The work demonstrates practical gains via minor code changes guided by probing insights, and positions probing as a valuable tool for improving transparency and design in NCO models.

Abstract

Neural combinatorial optimization (NCO) has achieved remarkable performance, yet its learned model representations and decision rationale remain a black box. This impedes both academic research and practical deployment, since researchers and stakeholders require deeper insights into NCO models. In this paper, we take the first critical step towards interpreting NCO models by investigating their representations through various probing tasks. Moreover, we introduce a novel probing tool named Coefficient Significance Probing (CS-Probing) to enable deeper analysis of NCO representations by examining the coefficients and statistical significance during probing. Extensive experiments and analysis reveal that NCO models encode low-level information essential for solution construction, while capturing high-level knowledge to facilitate better decisions. Using CS-Probing, we find that prevalent NCO models impose varying inductive biases on their learned representations, uncover direct evidence related to model generalization, and identify key embedding dimensions associated with specific knowledge. These insights can be potentially translated into practice, for example, with minor code modifications, we improve the generalization of the analyzed model. Our work represents a first systematic attempt to interpret black-box NCO models, showcasing probing as a promising tool for analyzing their internal mechanisms and revealing insights for the NCO community. The source code is publicly available.

Figures (16)

• Figure 1: Two groups of NCO model architectures: (a) represents the HELD structure, as seen in AM and POMO, while (b) represents the LEHD structure. The red arrows in the figure indicate the positions where we probe the model, extracting the embeddings.
• Figure 2: Probing results across different layers.
• Figure 3: Probing results during NCO models training.
• Figure 4: Heatmaps of node embeddings. The x-axis represents different embedding dimensions, and the y-axis represents the instances.
• Figure 5: Coefficients of probing models for two TSP-related probing tasks across all NCO models.