Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the Effect of Network Pruning on Performance and Interpretability

Jonathan von Rad, Florian Seuffert

TL;DR

This work investigates how pruning techniques influence performance and interpretability in GoogLeNet, comparing unstructured, structured, and connection sparsity approaches under iterative and one-shot retraining. It finds that Connection Sparsity can achieve substantial parameter reduction while maintaining or slightly improving accuracy, especially with iterative retraining, though it is computationally demanding. The study uses the Mechanistic Interpretability Score (MIS) to assess interpretability and reveals that MIS does not reliably track pruning level or predictive accuracy, with high MIS possible even for low-accuracy models. Overall, the results highlight a practical path to compact, performant networks via input-channel pruning, while also calling into question current interpretability metrics and the need for more robust measures tied to decision quality.

Abstract

Deep Neural Networks (DNNs) are often over-parameterized for their tasks and can be compressed quite drastically by removing weights, a process called pruning. We investigate the impact of different pruning techniques on the classification performance and interpretability of GoogLeNet. We systematically apply unstructured and structured pruning, as well as connection sparsity (pruning of input weights) methods to the network and analyze the outcomes regarding the network's performance on the validation set of ImageNet. We also compare different retraining strategies, such as iterative pruning and one-shot pruning. We find that with sufficient retraining epochs, the performance of the networks can approximate the performance of the default GoogLeNet - and even surpass it in some cases. To assess interpretability, we employ the Mechanistic Interpretability Score (MIS) developed by Zimmermann et al. . Our experiments reveal that there is no significant relationship between interpretability and pruning rate when using MIS as a measure. Additionally, we observe that networks with extremely low accuracy can still achieve high MIS scores, suggesting that the MIS may not always align with intuitive notions of interpretability, such as understanding the basis of correct decisions.

Investigating the Effect of Network Pruning on Performance and Interpretability

TL;DR

This work investigates how pruning techniques influence performance and interpretability in GoogLeNet, comparing unstructured, structured, and connection sparsity approaches under iterative and one-shot retraining. It finds that Connection Sparsity can achieve substantial parameter reduction while maintaining or slightly improving accuracy, especially with iterative retraining, though it is computationally demanding. The study uses the Mechanistic Interpretability Score (MIS) to assess interpretability and reveals that MIS does not reliably track pruning level or predictive accuracy, with high MIS possible even for low-accuracy models. Overall, the results highlight a practical path to compact, performant networks via input-channel pruning, while also calling into question current interpretability metrics and the need for more robust measures tied to decision quality.

Abstract

Deep Neural Networks (DNNs) are often over-parameterized for their tasks and can be compressed quite drastically by removing weights, a process called pruning. We investigate the impact of different pruning techniques on the classification performance and interpretability of GoogLeNet. We systematically apply unstructured and structured pruning, as well as connection sparsity (pruning of input weights) methods to the network and analyze the outcomes regarding the network's performance on the validation set of ImageNet. We also compare different retraining strategies, such as iterative pruning and one-shot pruning. We find that with sufficient retraining epochs, the performance of the networks can approximate the performance of the default GoogLeNet - and even surpass it in some cases. To assess interpretability, we employ the Mechanistic Interpretability Score (MIS) developed by Zimmermann et al. . Our experiments reveal that there is no significant relationship between interpretability and pruning rate when using MIS as a measure. Additionally, we observe that networks with extremely low accuracy can still achieve high MIS scores, suggesting that the MIS may not always align with intuitive notions of interpretability, such as understanding the basis of correct decisions.
Paper Structure (9 sections, 6 figures)

This paper contains 9 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Effect on Performance
  3. Pruning Strategies
  4. Iterative vs. One-Shot Pruning
  5. Results
  6. Effect on Interpretability
  7. Mechanistic Interpretability Score (MIS)
  8. Considerations Regarding MIS
  9. Conclusion

Figures (6)

  • Figure 1: Pruning Rate vs. Accuracy for Global Unstructured Pruning (L1 norm) without fine-tuning.
  • Figure 2: Validation Accuracy Over 60 Epochs After Unstructured Pruning 89% of the Network (One-Shot Retraining).
  • Figure 3: Accuracy vs. Pruning Rate: Iterative Pruning with Fine-Tuning vs. One-Shot Pruning.
  • Figure 4: Summary of Pruning Methods and Their Impact on Accuracy. Black lines represent models without fine-tuning.
  • Figure 5: Average MIS Confidence vs. Pruning Rate. The black lines represent models without fine-tuning and therefore with near-random predictions.
  • ...and 1 more figures