Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers

Siyu Zhang

Abstract

Sparse neural networks are often hypothesized to be more interpretable than dense models, motivated by findings that weight sparsity can produce compact circuits in language models. However, it remains unclear whether structural sparsity itself leads to improved semantic interpretability. In this work, we systematically evaluate the relationship between weight sparsity and interpretability in Vision Transformers using DeiT-III B/16 models pruned with Wanda. To assess interpretability comprehensively, we introduce \textbf{IMPACT}, a multi-level framework that evaluates interpretability across four complementary levels: neurons, layer representations, task circuits, and model-level attribution. Layer representations are analyzed using BatchTopK sparse autoencoders, circuits are extracted via learnable node masking, and explanations are evaluated with transformer attribution using insertion and deletion metrics. Our results reveal a clear structural effect but limited interpretability gains. Sparse models produce circuits with approximately $2.5\times$ fewer edges than dense models, yet the fraction of active nodes remains similar or higher, indicating that pruning redistributes computation rather than isolating simpler functional modules. Consistent with this observation, sparse models show no systematic improvements in neuron-level selectivity, SAE feature interpretability, or attribution faithfulness. These findings suggest that structural sparsity alone does not reliably yield more interpretable vision models, highlighting the importance of evaluation frameworks that assess interpretability beyond circuit compactness.

Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers

Abstract

Sparse neural networks are often hypothesized to be more interpretable than dense models, motivated by findings that weight sparsity can produce compact circuits in language models. However, it remains unclear whether structural sparsity itself leads to improved semantic interpretability. In this work, we systematically evaluate the relationship between weight sparsity and interpretability in Vision Transformers using DeiT-III B/16 models pruned with Wanda. To assess interpretability comprehensively, we introduce \textbf{IMPACT}, a multi-level framework that evaluates interpretability across four complementary levels: neurons, layer representations, task circuits, and model-level attribution. Layer representations are analyzed using BatchTopK sparse autoencoders, circuits are extracted via learnable node masking, and explanations are evaluated with transformer attribution using insertion and deletion metrics. Our results reveal a clear structural effect but limited interpretability gains. Sparse models produce circuits with approximately fewer edges than dense models, yet the fraction of active nodes remains similar or higher, indicating that pruning redistributes computation rather than isolating simpler functional modules. Consistent with this observation, sparse models show no systematic improvements in neuron-level selectivity, SAE feature interpretability, or attribution faithfulness. These findings suggest that structural sparsity alone does not reliably yield more interpretable vision models, highlighting the importance of evaluation frameworks that assess interpretability beyond circuit compactness.
Paper Structure (63 sections, 15 equations, 4 figures, 7 tables)

This paper contains 63 sections, 15 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Mechanistic Interpretability and Circuits.
  5. Interpretability in Vision Transformers.
  6. Superposition and Sparse Autoencoders.
  7. Weight Pruning and Interpretability.
  8. Transformer Attribution.
  9. Methods
  10. Sparse Model Construction
  11. Neuron-Level Interpretability
  12. Ablation Impact.
  13. Selectivity.
  14. Class Variance.
  15. Label Entropy.
  16. ...and 48 more sections

Figures (4)

  • Figure 1: The IMPACT (Interpretability Multi-level Pipeline for Assessing Computational Transparency) framework used to evaluate interpretability across four levels of analysis.
  • Figure 2: Neuron-level interpretability metrics across layers (columns) and sparsity levels (colors). Each distribution spans all 1,000 ImageNet categories.
  • Figure 3: SAE feature interpretability for block 11 across sparsity levels.
  • Figure 4: Visual attribution comparing the dense DeiT3-B/16 model against models with sparsities of 0.1, 0.3, 0.5, and 0.7.