Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack

Kuo Gai, Sicong Wang, Shihua Zhang

TL;DR

This paper tackles adversarial vulnerability in deep networks by marrying optimal transport theory with convex integration to enforce local Lipschitz continuity without sacrificing data fit. OTAD first learns a discrete OT map from data to features using ResNet or Transformer backbones and then robustly interpolates this map via a convex integration problem, solved exactly by a QCP or approximated by a CIP-net. The approach yields superior robustness across diverse datasets (images, single-cell data, industrial data) and scales to complex architectures, with OTAD-T and OTAD-T-NN accelerating inference through neural surrogates. The work demonstrates that leveraging the regularity of OT maps provides a principled, effective defense beyond conventional adversarial training and Lipschitz-based methods, and outlines practical paths for further enhancement and broader applicability.

Abstract

Deep neural networks (DNNs) are vulnerable to small adversarial perturbations of the inputs, posing a significant challenge to their reliability and robustness. Empirical methods such as adversarial training can defend against particular attacks but remain vulnerable to more powerful attacks. Alternatively, Lipschitz networks provide certified robustness to unseen perturbations but lack sufficient expressive power. To harness the advantages of both approaches, we design a novel two-step Optimal Transport induced Adversarial Defense (OTAD) model that can fit the training data accurately while preserving the local Lipschitz continuity. First, we train a DNN with a regularizer derived from optimal transport theory, yielding a discrete optimal transport map linking data to its features. By leveraging the map's inherent regularity, we interpolate the map by solving the convex integration problem (CIP) to guarantee the local Lipschitz property. OTAD is extensible to diverse architectures of ResNet and Transformer, making it suitable for complex data. For efficient computation, the CIP can be solved through training neural networks. OTAD opens a novel avenue for developing reliable and secure deep learning systems through the regularity of optimal transport maps. Empirical results demonstrate that OTAD can outperform other robust models on diverse datasets.

OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack

TL;DR

This paper tackles adversarial vulnerability in deep networks by marrying optimal transport theory with convex integration to enforce local Lipschitz continuity without sacrificing data fit. OTAD first learns a discrete OT map from data to features using ResNet or Transformer backbones and then robustly interpolates this map via a convex integration problem, solved exactly by a QCP or approximated by a CIP-net. The approach yields superior robustness across diverse datasets (images, single-cell data, industrial data) and scales to complex architectures, with OTAD-T and OTAD-T-NN accelerating inference through neural surrogates. The work demonstrates that leveraging the regularity of OT maps provides a principled, effective defense beyond conventional adversarial training and Lipschitz-based methods, and outlines practical paths for further enhancement and broader applicability.

Abstract

Deep neural networks (DNNs) are vulnerable to small adversarial perturbations of the inputs, posing a significant challenge to their reliability and robustness. Empirical methods such as adversarial training can defend against particular attacks but remain vulnerable to more powerful attacks. Alternatively, Lipschitz networks provide certified robustness to unseen perturbations but lack sufficient expressive power. To harness the advantages of both approaches, we design a novel two-step Optimal Transport induced Adversarial Defense (OTAD) model that can fit the training data accurately while preserving the local Lipschitz continuity. First, we train a DNN with a regularizer derived from optimal transport theory, yielding a discrete optimal transport map linking data to its features. By leveraging the map's inherent regularity, we interpolate the map by solving the convex integration problem (CIP) to guarantee the local Lipschitz property. OTAD is extensible to diverse architectures of ResNet and Transformer, making it suitable for complex data. For efficient computation, the CIP can be solved through training neural networks. OTAD opens a novel avenue for developing reliable and secure deep learning systems through the regularity of optimal transport maps. Empirical results demonstrate that OTAD can outperform other robust models on diverse datasets.
Paper Structure (27 sections, 2 theorems, 32 equations, 2 figures, 14 tables, 2 algorithms)

This paper contains 27 sections, 2 theorems, 32 equations, 2 figures, 14 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Adversarial training
  4. Adversarial purification
  5. Lipschitz networks and random smoothing
  6. Method background
  7. Optimal transport
  8. Regularity of OT maps
  9. OTAD
  10. ResNet-based OTAD
  11. Transformer-based OTAD
  12. Neural network for CIP
  13. Finding better neighbors through metric learning
  14. Experimental results
  15. Experimental setup
  16. ...and 12 more sections

Key Result

Theorem 1

Given a sequence $x_1,x_2,...,x_N \in \mathbb{R}^{D}$, the input $X = [x_1,x_2,\cdots,x_N]^T \in \mathbb{R}^{N \times D}$ is bounded by $\|X\|_F \leq M$. For $1 \leq r \leq R$, $R$ is the number of head, let $Q^{(r)}, K^{(r)}, V^{(r)} \in \mathbb{R}^{D \times D/R}$, and $W \in \mathbb{R}^{D \times D The multi-head self-attention $F$ is defined by where $f^{(r)}$ is single-head dot-product self-at

Figures (2)

  • Figure 1: Illustration of OTAD and its implementation. DNNs are vulnerable to small adversarial perturbations of the inputs. To classify the adversarial inputs accurately, OTAD replaces the inference process of DNN by solving a convex integration problem with a neighborhood set, which guarantees the local Lipschitz property. Furthermore, OTAD can adopt deep metric learning to find more similar neighborhood sets. For fast inference, the CIP can be solved by employing a neural network trained with the solution of the QCP problem.
  • Figure 2: Performance of OTAD with different hyperparameter $L-l$ against BPDA + PGD ($\epsilon = 3$). (a) The increasing disparity between standard and robust accuracy as $L-l$ varies. (b) The local Lipschitz constant of OTAD (OTAD Lip) increases with $L-l$. (c) The relative error increases with increasing $L-l$.

Theorems & Definitions (4)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Lemma 1