Dense Associative Memory is Robust to Adversarial Inputs

TL;DR

This work tackles the vulnerability of classical deep nets to rubbish images and adversarial perturbations by studying Dense Associative Memory (DAM) models with higher-order energy functions. It shows that when the interaction power $n$ is large, DAM minima become semantically meaningful, decision-boundary deformations are human-interpretable, and adversarial examples from low-$n$ models do not transfer to high-$n$ DAMs, aligning more closely with human perception. The results further suggest a security-oriented role for high-$n$ DAMs in detecting adversarial inputs, while also acknowledging that simply replacing ReLUs with higher-degree rectified polynomials in standard nets is not enough. The paper proposes extensions such as adversarial training and convolutional adaptations to broaden applicability and robustness, with implications for both theory and practical defense against adversarial threats.

Abstract

Deep neural networks (DNN) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation, so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNN and humans classify patterns, and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our paper examines these questions within the framework of Dense Associative Memory (DAM) models. These models are defined by the energy function, with higher order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units (ReLU), fail to transfer to and fool the models with higher order interactions. This opens up a possibility to use higher order models for detecting and stopping malicious adversarial attacks. The presented results suggest that DAM with higher order energy functions are closer to human visual perception than DNN with ReLUs.

Figures (7)

• Figure 1: Architecture of the neural network. Visible neurons $v_i$ are equal to the intensities of the pixels, so that the $8$-bit intensities are linearly mapped onto the segment $[-1, +1]$. There are $N=784$ visible neurons $v_i$, and $N_c=10$ classification neurons, which are initialized in the "off" state $x_\alpha=-1$ and then updated once to the final state $Y_\alpha$ using (\ref{['update_rule']}). The model has $K=2000$ memories, with visible parts $\xi^\mu_i$ and recognition part $\xi^\mu_\alpha$, index $\mu=1...K$. Parameter $\beta$ regulates the slope of the $\tanh$ function.
• Figure 2: Feature to prototype transition for powers of the energy function $n=2, 3, 20, 30$. For each model 25 randomly selected feature (or prototype) detectors $\xi^\mu_i$ are shown. The value of the $i$-th element of a detector is plotted in the location of the $i$-th pixel to which it couples in the update rule (\ref{['update_rule']}). The weights are normalized so that $\-1\leq \xi^\mu_i \leq 1$. The color code is explained by the color bar.
• Figure 3: Examples of images generated from a gaussian noise ${\cal N} (0,0.1)$ by following the gradient of the 10 objective functions corresponding to 10 digit classes for model (\ref{['update_rule']}) with $n=2,3,20,30$. For $n=2$ and $n=3$ each final image is labeled. For $n=20$ and $n=30$ the labels are clear from the images. Throughout this paper we use grayscale intensities for representation of the image plane. They should not be confused with the color images, like the one in Fig.\ref{['feature_to_prototype']}, which shows the network's weights.
• Figure 4: (Top panel) Logarithm of the top-choice and second-choice objective functions as the iterative dynamics following the negative gradient of the second-choice objective function (red) progresses. The crossing point defines the decision boundary. (Bottom panel) A set of triplets of images for $n=2, 3, 20, 30$, three per model. The first image in each triplet is a natural image from the dataset, the middle image is an artificial image corresponding to the crossing point of the two objective functions (decision boundary), the third image corresponds to the final point of the iterative dynamics, when the second-choice objective function (red curve) reaches zero.
• Figure 5: Transfer table of adversarial examples. The MNIST test set was used to construct four test sets of adversarial images, poised one step behind the decision boundary, for $n=2, 3, 20, 30$. These four datasets were cross-classified by the same four models. Each dataset has 10000 images. The number at the intersection of the $i$-th row and the $j$-th column is the error rate of the $j$-th model on the adversarial dataset constructed using the $i$-th model. On the right the error rates of the four models on the original MNIST test set.