Optimization with Trained Machine Learning Models Embedded

TL;DR

Embedding trained ML models into deterministic global optimization creates large nonlinear programs but also exploits recurring structures, such as network layers, for specialized reformulations. The chapter compares reduced-space and full-space formulations, addresses nonsmooth activations and validity-domain constraints, and surveys four ML families—convex region surrogates, Gaussian processes, decision tree ensembles, and artificial neural networks—for embedding. It highlights MILP encodings (e.g., for ReLU and tree ensembles), hull-based relaxations, and domain-constraint strategies as key tools, while noting that RS vs. FS trade-offs and performance depend on problem class and tooling. Overall, the work provides a framework to enable scalable, provably global optimization with ML embeddings and points to the need for continued advances to keep pace with evolving architectures.

Abstract

Trained ML models are commonly embedded in optimization problems. In many cases, this leads to large-scale NLPs that are difficult to solve to global optimality. While ML models frequently lead to large problems, they also exhibit homogeneous structures and repeating patterns (e.g., layers in ANNs). Thus, specialized solution strategies can be used for large problem classes. Recently, there have been some promising works proposing specialized reformulations using mixed-integer programming or reduced space formulations. However, further work is needed to develop more efficient solution approaches and keep up with the rapid development of new ML model architectures.

Figures (1)

• Figure 1: Overview of ML models ordered by increasing complexity. Repreinted from Schweidtmann et al. (2021) schweidtmann2021machine (Figure licensed under Creative Commons CC BY).