Optimization over Trained Neural Networks: Going Large with Gradient-Based Algorithms

Abstract

When optimizing a nonlinear objective, one can employ a neural network as a surrogate for the nonlinear function. However, the resulting optimization model can be time-consuming to solve globally with exact methods. As a result, local search that exploits the neural-network structure has been employed to find good solutions within a reasonable time limit. For such methods, a lower per-iteration cost is advantageous when solving larger models. The contribution of this paper is two-fold. First, we propose a gradient-based algorithm with lower per-iteration cost than existing methods. Second, we further adapt this algorithm to exploit the piecewise-linear structure of neural networks that use Rectified Linear Units (ReLUs). In line with prior research, our methods become competitive with -- and then dominant over -- other local search methods as the optimization models become larger.

Figures (8)

• Figure 1: Top: Visual description of neural network used as example. Bottom left: Lines partitioning the space based on what inputs produce a positive output for each neuron, with the arrow pointing to the positive side, the length of the arrow proportional to the magnitude of the parameters, and the arrow label denoting the influence on $y$. Bottom right: Contour plots of $y$ over the lines associated with neural activations.
• Figure 2: Left: Solutions found by MILP Walk from the initial solution $A = (0.23, 0.636)$ until convergence. Right: Solutions found by LP Walk from $A$ until convergence.
• Figure 3: Left: First solutions found by Gradient Walk from the initial solution $A = (0.23, 0.636)$. Right: Next solutions found over a narrower region of the input space.
• Figure 4: Comparison of algorithm performance over all instances by varying time limit.
• Figure 5: Comparison of algorithm performance by varying input size in 120-minute runs.