Large Neighborhood Search meets Iterative Neural Constraint Heuristics

Abstract

Neural networks are being increasingly used as heuristics for constraint satisfaction. These neural methods are often recurrent, learning to iteratively refine candidate assignments. In this work, we make explicit the connection between such iterative neural heuristics and Large Neighborhood Search (LNS), and adapt an existing neural constraint satisfaction method-ConsFormer-into an LNS procedure. We decompose the resulting neural LNS into two standard components: the destroy and repair operators. On the destroy side, we instantiate several classical heuristics and introduce novel prediction-guided operators that exploit the model's internal scores to select neighborhoods. On the repair side, we utilize ConsFormer as a neural repair operator and compare the original sampling-based decoder to a greedy decoder that selects the most likely assignments. Through an empirical study on Sudoku, Graph Coloring, and MaxCut, we find that adapting the neural heuristic to an LNS procedure yields substantial gains over its vanilla settings and improves its competitiveness with classical and neural baselines. We further observe consistent design patterns across tasks: stochastic destroy operators outperform greedy ones, while greedy repair is more effective than sampling-based repair for finding a single high-quality feasible assignment. These findings highlight LNS as a useful lens and design framework for structuring and improving iterative neural approaches.

Figures (4)

• Figure 1: Comparison between the original ConsFormer update and our ConsFormer-LNS. Top: ConsFormer takes the current assignment $\mathbf{x}^{(t)}$, selects a random subset of variables, applies the Transformer, and samples a new assignment $\mathbf{x}^{(t+1)}$. Bottom: In ConsFormer-LNS, a destroy operator selects variables to modify, and the Transformer acts as part of a neural repair operator that proposes the next assignment $\mathbf{x}^{(t+1)}$.
• Figure 2: Sudoku instances solved (%) over LNS steps for the baseline random destroy and the best-performing classical (stochastic worst removal) and prediction-guided (stochastic confidence-margin removal) destroy operators. Both heuristics outperform random removal, with confidence-margin removal improving fastest early on and stochastic worst removal eventually achieving the highest solved percentage.
• Figure 3: Cell accuracy for Sudoku over iterations for greedy (blue) and stochastic (orange) variants of Worst and Confidence destroy operators. Greedy variants plateau early, while stochastic variants continue to improve and reach substantially higher accuracies.
• Figure 4: Sudoku constraint accuracy on OOD Test dataset over LNS steps for greedy (blue) and sampling-based (orange) repair under Random and two best-performing destroy operators (Stochastic Confidence-margin, Stochastic Worst). Split violin plots show the distribution of per-instance constraint satisfaction at selected steps, while the lines on the right axis show the cumulative fraction of instances solved. Greedy repair achieves higher final solved percentages but exhibits a wider tail of unsolved instances, whereas sampling attains higher average constraint accuracy at intermediate steps.