Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization

Daniel Kuelbs, Sanjay Lall, Mert Pilanci

TL;DR

A convex semidefinite program (SDP) is devised for adversarial training of two-layer polynomial activation networks and it is proved that the convex SDP achieves the same globally optimal solution as its nonconvex counterpart.

Abstract

Training neural networks which are robust to adversarial attacks remains an important problem in deep learning, especially as heavily overparameterized models are adopted in safety-critical settings. Drawing from recent work which reformulates the training problems for two-layer ReLU and polynomial activation networks as convex programs, we devise a convex semidefinite program (SDP) for adversarial training of two-layer polynomial activation networks and prove that the convex SDP achieves the same globally optimal solution as its nonconvex counterpart. The convex SDP is observed to improve robust test accuracy against $\ell_\infty$ attacks relative to the original convex training formulation on multiple datasets. Additionally, we present scalable implementations of adversarial training for two-layer polynomial and ReLU networks which are compatible with standard machine learning libraries and GPU acceleration. Leveraging these implementations, we retrain the final two fully connected layers of a Pre-Activation ResNet-18 model on the CIFAR-10 dataset with both polynomial and ReLU activations. The two `robustified' models achieve significantly higher robust test accuracies against $\ell_\infty$ attacks than a Pre-Activation ResNet-18 model trained with sharpness-aware minimization, demonstrating the practical utility of convex adversarial training on large-scale problems.

Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization

TL;DR

A convex semidefinite program (SDP) is devised for adversarial training of two-layer polynomial activation networks and it is proved that the convex SDP achieves the same globally optimal solution as its nonconvex counterpart.

Abstract

Training neural networks which are robust to adversarial attacks remains an important problem in deep learning, especially as heavily overparameterized models are adopted in safety-critical settings. Drawing from recent work which reformulates the training problems for two-layer ReLU and polynomial activation networks as convex programs, we devise a convex semidefinite program (SDP) for adversarial training of two-layer polynomial activation networks and prove that the convex SDP achieves the same globally optimal solution as its nonconvex counterpart. The convex SDP is observed to improve robust test accuracy against attacks relative to the original convex training formulation on multiple datasets. Additionally, we present scalable implementations of adversarial training for two-layer polynomial and ReLU networks which are compatible with standard machine learning libraries and GPU acceleration. Leveraging these implementations, we retrain the final two fully connected layers of a Pre-Activation ResNet-18 model on the CIFAR-10 dataset with both polynomial and ReLU activations. The two `robustified' models achieve significantly higher robust test accuracies against attacks than a Pre-Activation ResNet-18 model trained with sharpness-aware minimization, demonstrating the practical utility of convex adversarial training on large-scale problems.
Paper Structure (22 sections, 8 theorems, 45 equations, 2 figures, 2 tables)

This paper contains 22 sections, 8 theorems, 45 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Paper Organization & Notation
  4. Standard Convex Training for Two-Layer Neural Networks
  5. Polynomial Activation Networks
  6. ReLU Activation Networks
  7. Convex Adversarial Training for Two-Layer Neural Networks
  8. Polynomial Activation Networks
  9. ReLU Activation Networks
  10. Practical Implementation of Convex Adversarial ReLU Networks
  11. Numerical Results
  12. Exact Solutions on Small-Scale Problems
  13. Robust Image Classification
  14. Conclusion
  15. Acknowledgement
  16. ...and 7 more sections

Key Result

Theorem 2.1

The solution of the convex program (cvx_nonrobust) provides a globally optimal solution for the non-convex problem (noncvx_nonrobust) when the number of neurons $m$ satisfies $m\geq m^\star$, where $m^\star=\mathop{\mathrm{rank}}\nolimits{Z^\star} +\mathop{\mathrm{rank}}\nolimits{Z'^\star}$ and $Z^\

Figures (2)

  • Figure 1: We solve the optimizaton problem (\ref{['adversarial_sdp']}) on the Wisconsin Breast Cancer dataset for a range of $r$ values while holding $\beta=0.01$ constant. On the left, we plot average distance to the decision boundary of test examples against $r$. On the right, we plot robust accuracy against FGSM attacks of $\ell_\infty$ magnitude 0.8 for the adversarial and standard models. We additionally plot clean accuracy for the adversarial model.
  • Figure 2: Visualization of the results in Table \ref{['large-scale-results']}. Dashed lines indicate that the final convex layers were trained on a random $1\%$ subset of training data.

Theorems & Definitions (8)

  • Theorem 2.1: neuralsdp
  • Theorem 2.2: pmlr-v119-pilanci20a
  • Theorem 3.1: Our Result
  • Theorem A.1: S-procedure with inequality
  • Theorem A.2: S-procedure with equality
  • Lemma B.1: Max-min inequality
  • Lemma B.2
  • Corollary B.1