Towards Foundation Models for Mixed Integer Linear Programming

TL;DR

The paper addresses the lack of generalizable, foundation-model-style learning for MILP by introducing MILP-Evolve, a large-scale data-generation pipeline that creates diverse MILP classes using LLMs. It trains a single model per learning task (integrality-gap prediction, learning-to-branch, and Language-MILP contrastive learning) on this data, with a GNN-based encoder and attention to capture global structure, and demonstrates strong out-of-domain generalization and transfer to MIPLIB. Key findings show that diversity in training data greatly boosts generalization, and pretraining on MILP-Evolve accelerates convergence and performance on heterogeneous MILP benchmarks. The work lays groundwork for broadly applicable MILP foundation models and provides public code and data to spur further development in multi-class MILP learning and NLP-style alignment for optimization problems.

Abstract

Mixed Integer Linear Programming (MILP) is essential for modeling complex decision-making problems but faces challenges in computational tractability and requires expert formulation. Current deep learning approaches for MILP focus on specific problem classes and do not generalize to unseen classes. To address this shortcoming, we take a foundation model training approach, where we train a single deep learning model on a diverse set of MILP problems to generalize across problem classes. As existing datasets for MILP lack diversity and volume, we introduce MILP-Evolve, a novel LLM-based evolutionary framework that is capable of generating a large set of diverse MILP classes with an unlimited amount of instances. We study our methodology on three key learning tasks that capture diverse aspects of MILP: (1) integrality gap prediction, (2) learning to branch, and (3) a new task of aligning MILP instances with natural language descriptions. Our empirical results show that models trained on the data generated by MILP-Evolve achieve significant improvements on unseen problems, including MIPLIB benchmarks. Our work highlights the potential of moving towards a foundation model approach for MILP that can generalize to a broad range of MILP applications. Our code and data are publicly available at https://github.com/microsoft/OptiGuide.

Figures (13)

• Figure 1: Comparison of baseline and our model's performance across tasks, and visualization of MILP-Evolve generated MILP classes and seed MILP classes from prior work.
• Figure 2: Overview: Left—MILP-Evolve pipeline. We design an evolutionary pipeline to leverage a Large Language Model to generate a diverse set of MILP classes. Right—the learning tasks. We design three learning tasks capturing different aspects of MILPs for multi-class learning.
• Figure 3: An evolution chain for an auction problem consisting of two iterations. In the first iteration of the evolution, we add a constraint. The second iteration involves a crossover with the facility allocation problem. The details of code generated by LLM is in Appendix Figure \ref{['fig:class-code']}.
• Figure 4: Effect of Scaling Training Classes on Test Performance: Increasing training classes improves test performance across all learning tasks.
• Figure 5: Convergence Rate on the MIPLIB Dataset. We plot the test performance for models at different stages of the training process for transfer learning to MIPLIB. Initializing with Ours pre-trained model leads to significantly faster convergence than other baselines.