Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parseval Networks: Improving Robustness to Adversarial Examples

Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, Nicolas Usunier

TL;DR

Parseval networks address the vulnerability of deep nets to adversarial perturbations by constraining the Lipschitz constant at every hidden layer. The authors propose Parseval regularization to maintain weight matrices as approximate Parseval tight frames and replace simple sums with convex aggregations, enabling controlled per-layer sensitivity and a bounded overall Lipschitz constant. They introduce efficient training techniques, including one-step orthonormalization updates and simplex projections for aggregation coefficients, and validate the approach on MNIST, CIFAR-10/100, and SVHN with both fully connected nets and wide ResNets, showing competitive accuracy on clean data and enhanced robustness to adversarial noise. The findings also reveal faster convergence and more efficient use of network capacity, suggesting practical benefits for deploying robust models in real-world settings.

Abstract

We introduce Parseval networks, a form of deep neural networks in which the Lipschitz constant of linear, convolutional and aggregation layers is constrained to be smaller than 1. Parseval networks are empirically and theoretically motivated by an analysis of the robustness of the predictions made by deep neural networks when their input is subject to an adversarial perturbation. The most important feature of Parseval networks is to maintain weight matrices of linear and convolutional layers to be (approximately) Parseval tight frames, which are extensions of orthogonal matrices to non-square matrices. We describe how these constraints can be maintained efficiently during SGD. We show that Parseval networks match the state-of-the-art in terms of accuracy on CIFAR-10/100 and Street View House Numbers (SVHN) while being more robust than their vanilla counterpart against adversarial examples. Incidentally, Parseval networks also tend to train faster and make a better usage of the full capacity of the networks.

Parseval Networks: Improving Robustness to Adversarial Examples

TL;DR

Parseval networks address the vulnerability of deep nets to adversarial perturbations by constraining the Lipschitz constant at every hidden layer. The authors propose Parseval regularization to maintain weight matrices as approximate Parseval tight frames and replace simple sums with convex aggregations, enabling controlled per-layer sensitivity and a bounded overall Lipschitz constant. They introduce efficient training techniques, including one-step orthonormalization updates and simplex projections for aggregation coefficients, and validate the approach on MNIST, CIFAR-10/100, and SVHN with both fully connected nets and wide ResNets, showing competitive accuracy on clean data and enhanced robustness to adversarial noise. The findings also reveal faster convergence and more efficient use of network capacity, suggesting practical benefits for deploying robust models in real-world settings.

Abstract

We introduce Parseval networks, a form of deep neural networks in which the Lipschitz constant of linear, convolutional and aggregation layers is constrained to be smaller than 1. Parseval networks are empirically and theoretically motivated by an analysis of the robustness of the predictions made by deep neural networks when their input is subject to an adversarial perturbation. The most important feature of Parseval networks is to maintain weight matrices of linear and convolutional layers to be (approximately) Parseval tight frames, which are extensions of orthogonal matrices to non-square matrices. We describe how these constraints can be maintained efficiently during SGD. We show that Parseval networks match the state-of-the-art in terms of accuracy on CIFAR-10/100 and Street View House Numbers (SVHN) while being more robust than their vanilla counterpart against adversarial examples. Incidentally, Parseval networks also tend to train faster and make a better usage of the full capacity of the networks.

Paper Structure

This paper contains 29 sections, 19 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Robustness in Neural Networks
  4. Adversarial examples
  5. Generalization with adversarial examples
  6. Lipschitz constant of neural networks
  7. Linear layers:
  8. Convolutional layers:
  9. Aggregation layers/transfer functions:
  10. Parseval networks
  11. Parseval Regularization
  12. Orthonormality of weight matrices:
  13. Aggregation Layers:
  14. Parseval Training
  15. Orthonormality constraints:
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Sample images from the CIFAR-10 dataset, with corresponding adversarial examples. We show the original image and adversarial versions for SNR values of 24.7, 12.1 and 7.8.
  • Figure 2: Histograms of the singular values of the weight matrices at layers 1 and 4 of our network in CIFAR-10.
  • Figure 3: Performance of the models for various magnitudes of adversarial noise on MNIST (left) and CIFAR-10 (right).
  • Figure 4: Learning curves of Parseval wide resnets and Vanilla wide resnets on CIFAR-10 (right) and CIFAR-100 (left). Parseval networks converge faster than their vanilla counterpart.

Theorems & Definitions (1)

  • Remark 1: Orthogonality is required