Red Teaming Deep Neural Networks with Feature Synthesis Tools

TL;DR

The paper tackles the challenge of evaluating interpretability tools for debugging deep neural networks in out-of-distribution contexts by introducing Trojan Rediscovery, a benchmark with 12 trojans across patch, style, and natural-feature types. It systematically analyzes 16 attribution/saliency methods and 7 feature-synthesis methods, introducing two novel synthesis approaches, and assesses performance via human studies and automated CLIP evaluations. Results show dataset-based attribution tools often underperform simple baselines, while synthesis-based methods are more capable but still fall short of consistently rediscovering all trojans, especially style trojans; combining multiple tools yields the best practical diagnostic power. The work argues for task-based benchmarking and a toolbox-style interpretability approach, providing a public benchmark and methods that can guide future robust debugging and red-teaming of neural networks.

Abstract

Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using \emph{feature synthesis} methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools on debugging tasks. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when an interpretation is correct. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation. A website for this paper and its code is at https://benchmarking-interpretability.csail.mit.edu/

Figures (37)

• Figure 1: (a): Example visualizations from 9 feature synthesis tools attempting to discover a trojan trigger (see the top row of Table \ref{['tab:trojans']}) responsible for a bug in the model. Details are in Section \ref{['sec:benchmarking_feature_synthesis']}. (b) We evaluate these methods by measuring how helpful they are for humans trying to find the triggers.
• Figure 2: Example trojaned images of each type that we use. Patch trojans are triggered by a patch we insert in a source image. Style trojans are triggered by performing style transfer on an image. Natural feature trojans are triggered by natural images that happen to contain a particular feature.
• Figure 3: The first 7 rows show examples using methods from prior work for reconstructing the 'fork' natural feature trigger. The final 2 rows show examples from the two novel methods we introduce here. TABOR = TrojAn Backdoor inspection based on non-convex Optimization and Regularization guo2019tabor. Fourier feature visualization (FV) visualizes neurons using a fourier-space image parameterization olah2017feature while CPPN feature visualization uses a convolutional pattern producing network parameterization mordvintsev2018differentiable. Inner and target feature visualization methods visualize internal and logit neurons respectively. Adv. Patch = adversarial patch brown2017adversarial. RFLA-Perturb = robust feature-level adversaries produced by perturbing a generator as in casper2022robust. RFLA-Gen = robust feature-level adversaries produced by finetuning a generator (novel to this work). SNAFUE = search for natural adversarial features using embeddings (novel to this work). Details on all methods are in Section \ref{['sec:feature_synthesis_methods']} and Section \ref{['sec:new_feature_synthesis_methods']}.
• Figure 4: All results from human evaluators (left) showing the proportion out of 100 subjects who identified the correct trigger from an 8-option multiple choice question. Results from CLIP radford2021learning (right) showing the mean confidence on the correct trigger on an 8-way matching problem. Humans outperformed CLIP. "All" refers to using all visualizations from all 9 tools at once. A random-guess baseline achieves 0.125. Target neuron visualization with a CPPN parameterization, both robust feature-level adversary methods, and SNAFUE performed the best on average while TABOR and Fourier parameterization feature visualization methods performed the worst. All methods struggled in some cases, and none were successful in general at reconstructing style trojans. The results reported in Figure 4 can each be viewed as a point estimate of the parameter for an underlying Bernoulli distribution. As such, the standard error can be upper-bounded by $0.05$.
• Figure 5: Examples of trojaned images, ground truth attribution maps, and attribution maps from various methods, including an edge detector baseline. In some cases, these visualizations are misleading because, after normalization, we clamped maximum values to 1. This clamping distorts differences between large values. See Figure \ref{['fig:attribution']} for quantitative results.