In-distribution adversarial attacks on object recognition models using gradient-free search

TL;DR

Problem: neural networks exhibit adversarial misclassifications that reside inside the training distribution, challenging the notion that failures are solely due to data bias. Approach: CMA-Search, a gradient-free Covariance Matrix Adaptation Evolution Strategy, searches the vicinity of correctly classified samples across parametric and real-world data to locate in-distribution misclassifications. Findings: across parametric, rendered, and natural image datasets, in-distribution adversarial examples occur with high attack rates, including around 71% for camera perturbations, 42% for lighting, and over 50% on natural datasets like Co3D, with corroborating results on ImageNet; even shift-invariant architectures remain vulnerable. Significance: reveals security risks from in-distribution adversarial examples and provides CMA-Search as a benchmarking tool, along with mitigation directions such as boundary-focused sampling, initialization strategies, and multi-view test-time inference.

Abstract

Neural networks are susceptible to small perturbations in the form of 2D rotations and shifts, image crops, and even changes in object colors. Past works attribute these errors to dataset bias, claiming that models fail on these perturbed samples as they do not belong to the training data distribution. Here, we challenge this claim and present evidence of the widespread existence of perturbed images within the training data distribution, which networks fail to classify. We train models on data sampled from parametric distributions, then search inside this data distribution to find such in-distribution adversarial examples. This is done using our gradient-free evolution strategies (ES) based approach which we call CMA-Search. Despite training with a large-scale (0.5 million images), unbiased dataset of camera and light variations, CMA-Search can find a failure inside the data distribution in over 71% cases by perturbing the camera position. With lighting changes, CMA-Search finds misclassifications in 42% cases. These findings also extend to natural images from ImageNet and Co3D datasets. This phenomenon of in-distribution images presents a highly worrisome problem for artificial intelligence -- they bypass the need for a malicious agent to add engineered noise to induce an adversarial attack. All code, datasets, and demos are available at https://github.com/Spandan-Madan/in_distribution_adversarial_examples.

Figures (10)

• Figure 1: In-distribution adversarial attacks. (a) The data distribution (depicted in black) refers to the space of all camera and light variations. Typical adversarial examples are created by adding noise to the image, which may result in images out of the data distribution. CMA-Search finds failures inside the data distribution. (b) 3D scene setup for our rendered images with camera parameters illustrated. (c) Example images with camera and light variations.
• Figure 2: In-distribution adversarial attacks on parametric data sampled from high-dimensional, disjoint uniform distributions. (a) Attack rate measured using CMA-Search is $100\%$ for all models---there exists an in-distribution failure in the vicinity of every correctly classified sample. Models become robust beyond a critical dataset size, but the data needed scales poorly with dimensionality. (b) Average Euclidean distance between the starting point and the identified in-distribution adversarial sample increases as dataset size increases. (c) Church window plots depicting adversarial examples (red) located contiguously and in between the learned and ground-truth boundaries.
• Figure 3: In-distribution adversarial attacks in the camera parameter space. a) Sample in-distribution adversarial examples. Percentage of change in Camera Position and Camera Look At parameters needed to induce the misclassification are also reported. Attack rates are reported in Table \ref{['table:3d_vs_2d']}. (b) Distribution of camera parameters for in-distribution adversarial images. Unlike human vision, there were no clear patterns characterizing the camera and light conditions of misclassified images.
• Figure 4: In-distribution adversarial attacks on natural images. (a) Misclassifications in ImageNet caused by CMA-Search + novel view synthesis. Examples are presented for a ResNet model trained on ImageNet, and OpenAI’s CLIP model. (b) Sample errors for the Co3D dataset searched within $1-5$ frames of a correctly classified image. Attack rates for Co3D are reported in Table \ref{['table:co3d']}.
• Figure S1: Flowchart describing CMA-Search step-by-step.