Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On-Off Pattern Encoding and Path-Count Encoding as Deep Neural Network Representations

Euna Jung, Jaekeol Choi, EungGu Yun, Wonjong Rhee

TL;DR

This paper tackles understanding how DNN representations encode task-relevant information by introducing On-Off patterns and PathCount as two complementary probes. It defines precise notions for binary activation and active-path counting, and evaluates their impact by replacing layer activations and by correlating PathCount with representations. Key findings show upper layers rely heavily on these binary and path-based signals, with several networks maintaining performance under replacement. The work also demonstrates practical interpretability benefits by developing OnOff-CAM and PC-CAM, improving target localization with reduced information.

Abstract

Understanding the encoded representation of Deep Neural Networks (DNNs) has been a fundamental yet challenging objective. In this work, we focus on two possible directions for analyzing representations of DNNs by studying simple image classification tasks. Specifically, we consider \textit{On-Off pattern} and \textit{PathCount} for investigating how information is stored in deep representations. On-off pattern of a neuron is decided as `on' or `off' depending on whether the neuron's activation after ReLU is non-zero or zero. PathCount is the number of paths that transmit non-zero energy from the input to a neuron. We investigate how neurons in the network encodes information by replacing each layer's activation with On-Off pattern or PathCount and evaluating its effect on classification performance. We also examine correlation between representation and PathCount. Finally, we show a possible way to improve an existing DNN interpretation method, Class Activation Map (CAM), by directly utilizing On-Off or PathCount.

On-Off Pattern Encoding and Path-Count Encoding as Deep Neural Network Representations

TL;DR

This paper tackles understanding how DNN representations encode task-relevant information by introducing On-Off patterns and PathCount as two complementary probes. It defines precise notions for binary activation and active-path counting, and evaluates their impact by replacing layer activations and by correlating PathCount with representations. Key findings show upper layers rely heavily on these binary and path-based signals, with several networks maintaining performance under replacement. The work also demonstrates practical interpretability benefits by developing OnOff-CAM and PC-CAM, improving target localization with reduced information.

Abstract

Understanding the encoded representation of Deep Neural Networks (DNNs) has been a fundamental yet challenging objective. In this work, we focus on two possible directions for analyzing representations of DNNs by studying simple image classification tasks. Specifically, we consider \textit{On-Off pattern} and \textit{PathCount} for investigating how information is stored in deep representations. On-off pattern of a neuron is decided as `on' or `off' depending on whether the neuron's activation after ReLU is non-zero or zero. PathCount is the number of paths that transmit non-zero energy from the input to a neuron. We investigate how neurons in the network encodes information by replacing each layer's activation with On-Off pattern or PathCount and evaluating its effect on classification performance. We also examine correlation between representation and PathCount. Finally, we show a possible way to improve an existing DNN interpretation method, Class Activation Map (CAM), by directly utilizing On-Off or PathCount.
Paper Structure (14 sections, 6 equations, 4 figures, 4 tables)

This paper contains 14 sections, 6 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Works for understanding deep neural networks
  4. Works on On-Off pattern
  5. Works on paths
  6. Methodology
  7. On-Off pattern
  8. PathCount
  9. Experiment
  10. Experimental details
  11. Activation replacement with On-Off patterns and PathCount
  12. Correlation between representation and PathCount
  13. Modified CAM
  14. Conclusion

Figures (4)

  • Figure 1: Definition of a path. We define a path as a sequence of neurons between two neurons. In the figure, we show an example of an active path from one input element and one neuron at the last layer with an orange line. A blue line represents an inactive path that is not propagated through the network because one of the activations consisting of a path sequence is zero.
  • Figure 2: Classification accuracy when replacing layerwise activation with On-Off and PathCount. Image classification accuracy after the replacement of layerwise activation with On-Off and PathCount is presented. For analysis, we also present On-ratio, which means the ratio of neurons having positive values over the number of neurons in each layer.
  • Figure 3: An example of target-matching process on Tiled-ImageNet. The upper picture represents a correct case of target-matching on Tiled-ImageNet where the top-1 class label used for computing importance of each pixel matches the estimated target class based on Grad-CAM scores. The lower picture shows an incorrect case of target-matching where the estimated target class is 'English springer' while the real target class was 'Wool'. We evaluate the target-matching accuracy by computing the ratio of correct cases.
  • Figure 4: Degradation performance of Act-CAM, OnOff-CAM, and PC-CAM. We present the degradation performance of three modified CAMs on three networks, VGG16, GoogLeNet, and ResNet18 trained on ImageNet. Act-CAM that uses activation of the last convolution output shows the best degradation performance followed by OnOff-CAM and PC-CAM.