Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogate

Hung Pham, Aiden Ren, Ibrahim Tahir, Jiatai Tong, Thiago Serra

TL;DR

The paper tackles the tractability problem of optimization models embedding large neural networks by introducing a sparse surrogate approach obtained via network pruning. It develops MILP formulations for two constraint-learning tasks—network verification (adversarial perturbation detection) and function maximization—embedding either the original dense network or a pruned sparse surrogate. Across extensive experiments on MNIST and Fashion-MNIST, unstructured pruning without finetuning often yields notable speedups in verification, especially at high pruning rates, while finetuning can be beneficial mainly at very high sparsity but at runtime cost. The results demonstrate that sparse surrogates can provide cost-effective, scalable alternatives for constraint learning, with particularly strong impact on verification and more nuanced gains for maximization in larger networks.

Abstract

In constraint learning, we use a neural network as a surrogate for part of the constraints or of the objective function of an optimization model. However, the tractability of the resulting model is heavily influenced by the size of the neural network used as a surrogate. One way to obtain a more tractable surrogate is by pruning the neural network first. In this work, we consider how to approach the setting in which the neural network is actually a given: how can we solve an optimization model embedding a large and predetermined neural network? We propose surrogating the neural network itself by pruning it, which leads to a sparse and more tractable optimization model, for which we hope to still obtain good solutions with respect to the original neural network. For network verification and function maximization models, that indeed leads to better solutions within a time limit, especially -- and surprisingly -- if we skip the standard retraining step known as finetuning. Hence, a pruned network with worse inference for lack of finetuning can be a better surrogate.

Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogate

TL;DR

The paper tackles the tractability problem of optimization models embedding large neural networks by introducing a sparse surrogate approach obtained via network pruning. It develops MILP formulations for two constraint-learning tasks—network verification (adversarial perturbation detection) and function maximization—embedding either the original dense network or a pruned sparse surrogate. Across extensive experiments on MNIST and Fashion-MNIST, unstructured pruning without finetuning often yields notable speedups in verification, especially at high pruning rates, while finetuning can be beneficial mainly at very high sparsity but at runtime cost. The results demonstrate that sparse surrogates can provide cost-effective, scalable alternatives for constraint learning, with particularly strong impact on verification and more nuanced gains for maximization in larger networks.

Abstract

In constraint learning, we use a neural network as a surrogate for part of the constraints or of the objective function of an optimization model. However, the tractability of the resulting model is heavily influenced by the size of the neural network used as a surrogate. One way to obtain a more tractable surrogate is by pruning the neural network first. In this work, we consider how to approach the setting in which the neural network is actually a given: how can we solve an optimization model embedding a large and predetermined neural network? We propose surrogating the neural network itself by pruning it, which leads to a sparse and more tractable optimization model, for which we hope to still obtain good solutions with respect to the original neural network. For network verification and function maximization models, that indeed leads to better solutions within a time limit, especially -- and surprisingly -- if we skip the standard retraining step known as finetuning. Hence, a pruned network with worse inference for lack of finetuning can be a better surrogate.
Paper Structure (18 sections, 4 equations, 3 figures, 3 tables, 2 algorithms)

This paper contains 18 sections, 4 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. From Notation to Formulation
  3. Network Verification
  4. Function Maximization
  5. From Network Pruning to Sparse Surrogates
  6. Background
  7. The Sparse Surrogate Approach
  8. Network Verification
  9. Function Maximization
  10. Experiments for Network Verification
  11. Technical Details
  12. Results and Analysis
  13. Experiments for Function Maximization
  14. Technical Details
  15. Results and Analysis
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Time to find adversarial input to networks trained on MNIST by solving the verification problem directly ($x$ axis) or indirectly with Algorithm \ref{['alg:callback']} ($y$ axis) per pruning rate, use of finetuning, and inclusion of finetuning in runtime. Squares on top or (and) right sides indicate no adversarial input found for either (both). Ties are not counted.
  • Figure 2: Time to find adversarial input to networks on Fashion-MNIST by solving the verification problem directly ($x$ axis) or indirectly with Algorithm \ref{['alg:callback']} ($y$ axis) per pruning rate, use of finetuning, and inclusion of finetuning in runtime. Squares on top or (and) right sides indicate no adversarial input found for either (both). Ties are not counted.
  • Figure 3: Time to find adversarial input to networks on MNIST and Fashion-MNIST by solving the verification problem directly ($x$ axis) or indirectly with Algorithm \ref{['alg:callback']} ($y$ axis) per pruning rate for different forms of network pruning. Squares on top or (and) right sides indicate no adversarial input found for either (both). Ties are not counted.