Adversarial robustness of VAEs through the lens of local geometry

TL;DR

This paper demonstrates that an optimal way for an adversary to attack VAEs is to exploit a directional bias of a stochastic pullback metric tensor induced by the encoder and decoder networks.

Abstract

In an unsupervised attack on variational autoencoders (VAEs), an adversary finds a small perturbation in an input sample that significantly changes its latent space encoding, thereby compromising the reconstruction for a fixed decoder. A known reason for such vulnerability is the distortions in the latent space resulting from a mismatch between approximated latent posterior and a prior distribution. Consequently, a slight change in an input sample can move its encoding to a low/zero density region in the latent space resulting in an unconstrained generation. This paper demonstrates that an optimal way for an adversary to attack VAEs is to exploit a directional bias of a stochastic pullback metric tensor induced by the encoder and decoder networks. The pullback metric tensor of an encoder measures the change in infinitesimal latent volume from an input to a latent space. Thus, it can be viewed as a lens to analyse the effect of input perturbations leading to latent space distortions. We propose robustness evaluation scores using the eigenspectrum of a pullback metric tensor. Moreover, we empirically show that the scores correlate with the robustness parameter $β$ of the $β-$VAE. Since increasing $β$ also degrades reconstruction quality, we demonstrate a simple alternative using \textit{mixup} training to fill the empty regions in the latent space, thus improving robustness with improved reconstruction.

Figures (10)

• Figure 1: Left: Illustration that adversarial examples find non-smooth change in the latent encodings. A small perturbation in the input sample exploits a direction that maximally changes latent encoding by moving from high density to low/zero density region in the latent space. In this paper, we show an optimal perturbation can be found by moving along the eigendirection of the local pullback metric tensor of a data point. Right: A smooth mapping $f$ from the data manifold $\mathcal{M}$ to the latent manifold $\mathcal{N}$ induces pullback metrics on $\mathcal{M}$. The Jacobian $\boldsymbol{\mathsf{J}}_{f(x)} = \frac{\partial f}{\partial \boldsymbol{\mathsf{x}}}$ is a linear map from a tangent vector $y\in T_x M$ to a tangent vector $z\in T_{f(x)} N$ that induces a pullback Riemannian metric tensor $\boldsymbol{\mathsf{G}}_{\boldsymbol{\mathsf{x}}}=\boldsymbol{\mathsf{J}}_{f(x)}^T\boldsymbol{\mathsf{J}}_{f(x)}$. The determinant of metric tensor $\boldsymbol{\mathsf{G}}_{\boldsymbol{\mathsf{x}}}$ represents the change in infinitesimal volume element when projected to the latent space.
• Figure 2: Illustration of adversarial attack along the dominant eigenvector of a stochastic pullback metric tensor. The first two rows are the results of MNIST data, and the bottom two are on the FashionMNIST dataset. We evaluate the reconstruction for original images and its two corrupted versions with different step sizes $\delta_1=0.5233$ and $\delta_2=0.7443$. Moving along eigendirection doesn't affect the input image but significantly changes its reconstruction.
• Figure 3: The plot shows the change in the latent encoding of $\beta-$VAE (in terms of Euclidean distance) for different values of $\beta$ when moving along the dominant eigendirection of a pullback metric tensor $\hat{\boldsymbol{\mathsf{G}}_{\boldsymbol{\mathsf{x}}}}$ with different step size $\delta$. We can see for small $\beta$, the changes are of much higher magnitude compared to larger $\beta$, demonstrating that increasing the $\beta$ makes the latent space more smooth.
• Figure 4: Illustration of adversarial attack along the dominant eigenvector of a stochastic pullback metric tensor on CelebA dataset. We evaluate the reconstruction for original images and its two corrupted versions with different step sizes $\delta_1=0.5233$ and $\delta_2=0.7443$.
• Figure 5: Figure (a), on the left, we report the histogram of spectral radius and Von Neumann entropy (on test samples) for different values of $\beta$ in $\beta$-VAE. On the right, we report the average of two scores across test samples for an increasing value of $\beta$. We observe that increasing the value of $\beta$ suppresses the metric tensor's maximum eigenvalue, and the eigenspectrum distribution gets more isotropic. In the second row, we corrupt the test images along the top five eigendirections (denoted by $\lambda_1, \lambda_2, \lambda_3, \lambda_4, \text{ and } \lambda_5$) with an increasing step size for different values of $\beta$. The plots describe the average MSE across test samples. We observe that the average step size increases for a higher value of $\beta$. Increasing the value of $\beta$ reduces the posterior-prior gap, minimising distortion in the latent space. Figure (b) demonstrates similar observations on the FashionMNIST dataset.

Theorems & Definitions (7)

• Definition 2.1
• Definition 2.2
• Definition 2.3
• Definition 2.4
• Remark 3.1
• Theorem 3.2
• proof