A General Neural Backbone for Mixed-Integer Linear Optimization via Dual Attention

TL;DR

This work introduces a dual-attention backbone for mixed-integer linear optimization that treats variables and constraints as two interacting element types. By combining intra-type self-attention with inter-type cross-attention, the model overcomes the locality limits of traditional GNNs, enabling global information exchange and deeper representations. Across instance-, element-, and solving-state tasks, the approach consistently outperforms BGNN-based baselines on synthetic benchmarks and real-world MILP libraries, and mechanism analyses reveal stronger long-range dependencies, richer embeddings, and more confident predictions. The results suggest attention-based architectures as a powerful, general foundation for learning-enhanced MILP solvers and potentially other broad COPs.

Abstract

Mixed-integer linear programming (MILP), a widely used modeling framework for combinatorial optimization, are central to many scientific and engineering applications, yet remains computationally challenging at scale. Recent advances in deep learning address this challenge by representing MILP instances as variable-constraint bipartite graphs and applying graph neural networks (GNNs) to extract latent structural patterns and enhance solver efficiency. However, this architecture is inherently limited by the local-oriented mechanism, leading to restricted representation power and hindering neural approaches for MILP. Here we present an attention-driven neural architecture that learns expressive representations beyond the pure graph view. A dual-attention mechanism is designed to perform parallel self- and cross-attention over variables and constraints, enabling global information exchange and deeper representation learning. We apply this general backbone to various downstream tasks at the instance level, element level, and solving state level. Extensive experiments across widely used benchmarks show consistent improvements of our approach over state-of-the-art baselines, highlighting attention-based neural architectures as a powerful foundation for learning-enhanced mixed-integer linear optimization.

Figures (6)

• Figure 1: Overview of the proposed attention-driven architecture.
• Figure 2: Model Architecture and applications in downstream tasks. (a) The dual-attention mechanism (middle panel) consists of two intra-type self-attention modules (left panel) and two inter-type cross-attention modules (right panel) that work collectively in parallel to extract variable and constraint embeddings. (b) Applying our model to three distinct tasks on instance level (predicting instance feasibility and optimal objective value), element level (predicting values of binary variables), and solving state level (predicting the variable to branch at each B&B node).
• Figure 3: Average Primal Gap (PG) of each method as a function of solving time. Each panel visualizes the solving process of a dataset. The black solid line is Gurobi, while other solid and dashed lines correspond to our attention-driven architecture and the BGNN baseline, respectively, combined with two predict-and-search frameworks (PaS and Apollo). Using our model achieves consistently smaller PG and faster convergence across all problems, indicating improved optimization efficiency and better integration with Gurobi.
• Figure 4: Comparison of the performance in extracting deep representations. (a) and (e) depict the normalized embedding values of the BGNN and our model at the 1st and 5th layer. (b) and (f) depict the t-SNE visualization of embeddings at the 1st and 5th layer of BGNN and our model. (c) and (g) depict the decay of gradients in the BGNN and our model as the network deepens. (d) and (h) depict the mean Macro-F1 score with standard deviation of training BGNN and our model five times under different numbers of layers.
• Figure 5: Analysis of long-range dependency modeling in BGNN and our attention-based architecture. Gradient-based attribution is used to quantify the influence strength between a target variable and other nodes at different distances. Panels (a) and (b) show average variable-to-variable influence for BGNN and our model, while panels (c) and (d) show average variable-to-constraint influence. BGNN exhibits rapidly diminishing influence with increasing distance, while the attention-based model maintains non-negligible influence across farther hops. Insets report the cumulative influence contributed by nodes within and beyond three hops.