NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer

TL;DR

NEPENTHE introduces an entropy-guided unstructured pruning framework to reduce neural network depth by removing zero-entropy layers in rectifier-activated networks. It defines neuron- and layer-level entropy metrics, shows that unstructured pruning naturally lowers layer entropy, and uses an entropy-aware pruning budget to reallocate capacity toward higher-entropy layers, enabling layer removal with minimal performance loss. Evaluations across ResNet-18, MobileNet-V2, and Swin-T on CIFAR-10, Tiny ImageNet, and diverse DomainBed datasets demonstrate more removable layers and better accuracy retention than iterative magnitude pruning or Layer Folding in many settings. Limitations arise on ImageNet with already efficient architectures, where depth reduction is harder, highlighting the need for differentiable entropy proxies and further refinements for less over-parameterized tasks.

Abstract

While deep neural networks are highly effective at solving complex tasks, their computational demands can hinder their usefulness in real-time applications and with limited-resources systems. Besides, for many tasks it is known that these models are over-parametrized: neoteric works have broadly focused on reducing the width of these networks, rather than their depth. In this paper, we aim to reduce the depth of over-parametrized deep neural networks: we propose an eNtropy-basEd Pruning as a nEural Network depTH's rEducer (NEPENTHE) to alleviate deep neural networks' computational burden. Based on our theoretical finding, NEPENTHE focuses on un-structurally pruning connections in layers with low entropy to remove them entirely. We validate our approach on popular architectures such as MobileNet and Swin-T, showing that when encountering an over-parametrization regime, it can effectively linearize some layers (hence reducing the model's depth) with little to no performance loss. The code will be publicly available upon acceptance of the article.

Figures (3)

• Figure 1: In this work we show that the average neuron's entropy calculated at the layer scale reduces as we induce some sparsity in the model. The main challenge is to make the average neuron's entropy go to zero for some layers, as it will be possible to remove it.
• Figure 2: Distribution of a layer's parameters with magnitude pruning at threshold $t$ (a), pre-activation distribution at varying $t$ under the assumption of independence and centering of the Gaussian distributed input and layer's parameters (b), and entropy of the rectifier-activated neuron's output as a function of $t$ (c), all in the large $N$ limit.
• Figure 3: Distributions of neuron states per layer for ResNet-18 trained on CIFAR-10 pruned by NEPENTHE. In blue neurons having non-zero entropy, in orange always OFF, and in red always ON.