Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence

Frederik Pahde, Maximilian Dreyer, Leander Weber, Moritz Weckbecker, Christopher J. Anders, Thomas Wiegand, Wojciech Samek, Sebastian Lapuschkin

TL;DR

Concept Activation Vectors (CAVs) are used to model human-interpretable concepts in neural latent spaces. The authors show that filter-based CAVs, which optimize class separability, can diverge from the true concept direction due to distractor patterns, and they introduce pattern-based CAVs that isolate the concept signal via a regression-based approach. Pattern-CAVs yield higher alignment with ground-truth concept directions and improve reliability in TCAV-based sensitivity testing and ClArC artifact correction across ISIC2019, Pediatric Bone Age, and FunnyBirds with diverse architectures. The findings demonstrate that focusing on the concept signal rather than separability enhances precision, robustness to scaling/noise, and practical utility in explainability and model debugging. This work provides a concrete path toward more trustworthy, artifact-resilient explanations for deep neural networks.

Abstract

With a growing interest in understanding neural network prediction strategies, Concept Activation Vectors (CAVs) have emerged as a popular tool for modeling human-understandable concepts in the latent space. Commonly, CAVs are computed by leveraging linear classifiers optimizing the separability of latent representations of samples with and without a given concept. However, in this paper we show that such a separability-oriented computation leads to solutions, which may diverge from the actual goal of precisely modeling the concept direction. This discrepancy can be attributed to the significant influence of distractor directions, i.e., signals unrelated to the concept, which are picked up by filters (i.e., weights) of linear models to optimize class-separability. To address this, we introduce pattern-based CAVs, solely focussing on concept signals, thereby providing more accurate concept directions. We evaluate various CAV methods in terms of their alignment with the true concept direction and their impact on CAV applications, including concept sensitivity testing and model correction for shortcut behavior caused by data artifacts. We demonstrate the benefits of pattern-based CAVs using the Pediatric Bone Age, ISIC2019, and FunnyBirds datasets with VGG, ResNet, ReXNet, EfficientNet, and Vision Transformer as model architectures.

Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence

TL;DR

Concept Activation Vectors (CAVs) are used to model human-interpretable concepts in neural latent spaces. The authors show that filter-based CAVs, which optimize class separability, can diverge from the true concept direction due to distractor patterns, and they introduce pattern-based CAVs that isolate the concept signal via a regression-based approach. Pattern-CAVs yield higher alignment with ground-truth concept directions and improve reliability in TCAV-based sensitivity testing and ClArC artifact correction across ISIC2019, Pediatric Bone Age, and FunnyBirds with diverse architectures. The findings demonstrate that focusing on the concept signal rather than separability enhances precision, robustness to scaling/noise, and practical utility in explainability and model debugging. This work provides a concrete path toward more trustworthy, artifact-resilient explanations for deep neural networks.

Abstract

With a growing interest in understanding neural network prediction strategies, Concept Activation Vectors (CAVs) have emerged as a popular tool for modeling human-understandable concepts in the latent space. Commonly, CAVs are computed by leveraging linear classifiers optimizing the separability of latent representations of samples with and without a given concept. However, in this paper we show that such a separability-oriented computation leads to solutions, which may diverge from the actual goal of precisely modeling the concept direction. This discrepancy can be attributed to the significant influence of distractor directions, i.e., signals unrelated to the concept, which are picked up by filters (i.e., weights) of linear models to optimize class-separability. To address this, we introduce pattern-based CAVs, solely focussing on concept signals, thereby providing more accurate concept directions. We evaluate various CAV methods in terms of their alignment with the true concept direction and their impact on CAV applications, including concept sensitivity testing and model correction for shortcut behavior caused by data artifacts. We demonstrate the benefits of pattern-based CAVs using the Pediatric Bone Age, ISIC2019, and FunnyBirds datasets with VGG, ResNet, ReXNet, EfficientNet, and Vision Transformer as model architectures.
Paper Structure (53 sections, 3 theorems, 50 equations, 37 figures, 11 tables)

This paper contains 53 sections, 3 theorems, 50 equations, 37 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Estimating Signal of Concept Direction
  4. Filter-based CAV Computation
  5. Pattern-based
  6. 2D Toy Experiments
  7. Experiments
  8. Experiment Details
  9. Preciseness of Concept Representation
  10. How clean are CAVs qualitatively?
  11. CAV Alignment with True Concept Direction
  12. Impact of Directional Shifts on CAV Applications
  13. Testing with CAV
  14. TCAV in 2D Toy Experiment
  15. Controlled Experiment with FunnyBirds
  16. ...and 38 more sections

Key Result

Theorem B.1

Define the vector $\widetilde{\mathbf{w}}_{\lambda} = \lambda $ where $\widetilde{w}_2 = -\frac{\sin \tau \cos \tau}{\sigma^2+\cos^2\tau}$. Then $\widetilde{\mathbf{w}}_{\lambda}$ is the unique minimizer

Figures (37)

  • Figure 1: CAVs obtained from filters, i.e. weight vectors from linear classifiers, are optimized for class separability, but fail at precisely estimating concept signal directions. Left: Different CAV computation strategies are employed to estimate the "band-aid" concept, a confounding artifact in the ISIC2019 dataset. Right: Weaknesses of filter-based CAVs are apparent for simple transformations in a 2D toy experiment, where we scale concept features (x-axis) differently than other (e.g., distracting) features or rotate distracting directions. Only pattern-based CAVs precisely estimate the concept signal direction, while filter-based CAVs diverge to optimize class separability. Animated visualizations for these and additional 2D experiments can be found here: https://github.com/frederikpahde/pattern-cav/tree/main/animations.
  • Figure 2: Example for timestamp artifact inserted into ISIC2019 samples (left) and RelMax visualization for neurons (right) corresponding to the largest absolute values in filter- and pattern-, along with the Conv filter ID and the fraction of all (absolute) CAV values. While the filter- picks up noisy neurons, the pattern- uses neurons related to the relevant concept.
  • Figure 3: Left: Comparison of cosine similarity between and true concept direction (top) and concept separability (bottom), using filter- (SVM) and pattern-CAV for all Conv layers of VGG16 trained on ISIC2019, Bone Age, and FunnyBirds. While expectedly filter- have superior class-separability, pattern- have a better alignment with the true concept direction. Right: Cosine similarity between true concept direction $\mathbf{h}^{\text{gt}}_i$ and with different feature pre-processing methods fitted on the last Conv layer of VGG16 trained on ISIC2019 and Bone Age. Compared to filter-, pattern- has a higher alignment with $\mathbf{h}^{\text{gt}}_i$ and is invariant to feature pre-processing.
  • Figure 4: Left: 2D TCAV experiment with distractor rotated by $\tau=45^{\circ}$ with samples from class A (purple with concept, blue without concept) and class B (green). The model $f$ classifies between classes A and B. CAVs are fitted on samples with and without concept from class A. The pattern-CAV aligns with the concept direction, while the filter-CAV diverges to optimize class-separability. Right:$\text{TCAV}_{\text{sens}}$ for model $f$ plotted over distractor rotation $\tau$. Positive and negative values indicate a positive and negative influence of the concept direction and $0$ indicates insensitivity ($\text{TCAV}=0.5$).
  • Figure 5: Left:$\Delta\text{TCAV}$ (averaged over class-defining concepts) for different fitted on last Conv layers of VGG16, ResNet18, and EfficientNet-B0 trained on FunnyBirds. As models must use these concepts by experimental design, high scores are better. In contrast to filter-, pattern- achieve best scores for all models. Right: Concept-sensitivity maps, measured as element-wise product $\boldsymbol\nabla_{\mathbf{a}} \Tilde{f}(\mathbf{a}(\mathbf{x})) \odot \mathbf{h}$ using filter- and pattern- for three concepts with VGG16 and EfficientNet-B0. Results are shown for the last Conv layer, upsampled to input space dimensions. While pattern- precisely localize the concepts, filter- lead to noisy sensitivity maps.
  • ...and 32 more figures

Theorems & Definitions (6)

  • Theorem B.1
  • proof
  • Theorem B.2
  • proof
  • Theorem B.3
  • proof