Efficient Decoder Scaling Strategy for Neural Routing Solvers

TL;DR

A systematic study comparing two distinct strategies for the efficient allocation of parameters and compute resources in construction-based neural routing solvers, demonstrating that scaling depth yields superior performance gains to scaling width.

Abstract

Construction-based neural routing solvers, typically composed of an encoder and a decoder, have emerged as a promising approach for solving vehicle routing problems. While recent studies suggest that shifting parameters from the encoder to the decoder enhances performance, most works restrict the decoder size to 1-3M parameters, leaving the effects of scaling largely unexplored. To address this gap, we conduct a systematic study comparing two distinct strategies: scaling depth versus scaling width. We synthesize these strategies to construct a suite of 12 model configurations, spanning a parameter range from 1M to ~150M, and extensively evaluate their scaling behaviors across three critical dimensions: parameter efficiency, data efficiency, and compute efficiency. Our empirical results reveal that parameter count is insufficient to accurately predict the model performance, highlighting the critical and distinct roles of model depth (layer count) and width (embedding dimension). Crucially, we demonstrate that scaling depth yields superior performance gains to scaling width. Based on these findings, we provide and experimentally validate a set of design principles for the efficient allocation of parameters and compute resources to enhance the model performance.

Figures (12)

• Figure 1: Decoder Parameters Count vs. Optimality Gap. The results demonstrate that simply increasing model size does not guarantee performance gains. The green circles highlight depth-prioritized models (constructed by prioritizing depth scaling over width), which dominate at least one larger model.
• Figure 2: Scaling Law Analysis for Decoder-only Models. (a) Global Fit: Fitting a single power law (blue dashed line) to all models results in a poor fit ($R^2=0.794$) with a high average prediction error of $34.39\%$, indicating that parameter count alone is insufficient to predict performance. (b) Scaling Width vs. (c) Scaling Depth: We decouple the scaling behaviors by fixing one dimension. Comparing the scaling exponents ($\alpha_n$), we observe that scaling depth (c) yields significantly steeper slopes ($\alpha_n \approx 0.98 \text{--} 1.05$) compared to scaling width (b) ($\alpha_n \approx 0.24 \text{--} 0.40$). This quantitatively demonstrates that Depth-Prioritized Models exhibit a much faster rate of performance improvement than width-prioritized ones as parameters increase.
• Figure 3: Data Efficiency Analysis. The log-log plot compares the optimality gap against training data size across different architectures. Prioritizing depth yields superior data efficiency: deeper models (e.g., $D=42$) exhibit steeper scaling exponents (up to $0.71$) compared to wider models (e.g., $W=512$, exponent $0.55$).
• Figure 4: Compute Efficiency Analysis. Comparison of optimality gap vs. inference compute budget per instance (GFLOPs) for (a) width scaling and (b) depth scaling. Depth scaling demonstrates superior efficiency, achieving a significantly higher scaling exponent ($\alpha=1.71$ for $D=42$) compared to width scaling ($\alpha=1.34$ for $W=512$).
• Figure 5: Visualization of the performance gap against model parameters ($N$) in log scale for varying depth-to-width ratios ($A$). The consistent drop in the gap as lines shift from cyan (low $A$) to pink (high $A$) illustrates that prioritizing deeper, narrower architectures effectively reduces the optimality gap under constrained parameter budgets.