Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A constrained optimization approach to improve robustness of neural networks

Shudian Zhao, Jan Kronqvist

TL;DR

An efficient cutting-plane-based algorithm to iteratively solve the large-scale nonconvex optimization problem by approximating the feasible region through polyhedral cuts and balancing between robustness and accuracy is proposed.

Abstract

In this paper, we present a novel nonlinear programming-based approach to fine-tune pre-trained neural networks to improve robustness against adversarial attacks while maintaining high accuracy on clean data. Our method introduces adversary-correction constraints to ensure correct classification of adversarial data and minimizes changes to the model parameters. We propose an efficient cutting-plane-based algorithm to iteratively solve the large-scale nonconvex optimization problem by approximating the feasible region through polyhedral cuts and balancing between robustness and accuracy. Computational experiments on standard datasets such as MNIST and CIFAR10 demonstrate that the proposed approach significantly improves robustness, even with a very small set of adversarial data, while maintaining minimal impact on accuracy.

A constrained optimization approach to improve robustness of neural networks

TL;DR

An efficient cutting-plane-based algorithm to iteratively solve the large-scale nonconvex optimization problem by approximating the feasible region through polyhedral cuts and balancing between robustness and accuracy is proposed.

Abstract

In this paper, we present a novel nonlinear programming-based approach to fine-tune pre-trained neural networks to improve robustness against adversarial attacks while maintaining high accuracy on clean data. Our method introduces adversary-correction constraints to ensure correct classification of adversarial data and minimizes changes to the model parameters. We propose an efficient cutting-plane-based algorithm to iteratively solve the large-scale nonconvex optimization problem by approximating the feasible region through polyhedral cuts and balancing between robustness and accuracy. Computational experiments on standard datasets such as MNIST and CIFAR10 demonstrate that the proposed approach significantly improves robustness, even with a very small set of adversarial data, while maintaining minimal impact on accuracy.
Paper Structure (18 sections, 2 theorems, 28 equations, 4 figures, 5 tables, 2 algorithms)

This paper contains 18 sections, 2 theorems, 28 equations, 4 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Notation
  4. Robust training of Neural Networks
  5. The adversary-correction problem
  6. The cutting-planed based fine-turning algorithm
  7. Updating the linear approximation
  8. Considering small changes in the adversarial data
  9. Balancing between the adversary-correction and loss on training data
  10. Main algorithm
  11. Implementation details
  12. Computational experiments
  13. Linearized input perturbations
  14. Accuracy for MNIST
  15. Accuracy on CIFAR10
  16. ...and 3 more sections

Key Result

Theorem 1

There exists a finite set of adversarial data $\mathcal{X}_\text{adv}$, such that an optimal solution (if one exists) to the adversarial correction problem eq:robust-project_org is also a feasible solution to the robust fine-tuning problem eq:robust_training.

Figures (4)

  • Figure 1: Candidate solutions and their efficient front after 5 iterations with Algorithm \ref{['alg:model_update']}, on the CNN model that is presented in Section \ref{['sec:experiments']}. Note that we only ran 5 iterations to reduce the number of points for illustration purposes.
  • Figure 2: Improvements of robustness in percentage points with $|\mathcal{X}_{adv}|=\{10,20,30,40,50\}$ using Algorithm \ref{['alg:model_update']} with $M=2$.
  • Figure 3: The comparison between $\bar{\varepsilon} = \{0, 0.05 \}$ with $|\mathcal{X}_{adv}|=50$, and $\omega =0$.
  • Figure 4: This figure shows the total running time and the time spent on solving the QP problems \ref{['eq:qp_cut_iter']} (i.e., running time of Gurobi).

Theorems & Definitions (11)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4: $\varepsilon$-robust
  • Definition 5: Strictly more robust over $\mathcal{X}_{train}$
  • Theorem 1
  • proof
  • Definition 6: Pareto optimal
  • Definition 7: Efficient front / Pareto front
  • Proposition 1
  • ...and 1 more