Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Electrostatic Force Regularization for Neural Structured Pruning

Abdesselam Ferdi, Abdelmalik Taleb-Ahmed, Amir Nakib, Youcef Ferdi

TL;DR

A novel method is proposed that, for the first time, incorporates the concepts of charge and electrostatic force from physics into the training process of DCNNs, and simultaneously optimizes weights and ranks filter importance, all without the need for extensive fine-tuning.

Abstract

The demand for deploying deep convolutional neural networks (DCNNs) on resource-constrained devices for real-time applications remains substantial. However, existing state-of-the-art structured pruning methods often involve intricate implementations, require modifications to the original network architectures, and necessitate an extensive fine-tuning phase. To overcome these challenges, we propose a novel method that, for the first time, incorporates the concepts of charge and electrostatic force from physics into the training process of DCNNs. The magnitude of this force is directly proportional to the product of the charges of the convolution filter and the source filter, and inversely proportional to the square of the distance between them. We applied this electrostatic-like force to the convolution filters, either attracting filters with opposite charges toward non-zero weights or repelling filters with like charges toward zero weights. Consequently, filters subject to repulsive forces have their weights reduced to zero, enabling their removal, while the attractive forces preserve filters with significant weights that retain information. Unlike conventional methods, our approach is straightforward to implement, does not require any architectural modifications, and simultaneously optimizes weights and ranks filter importance, all without the need for extensive fine-tuning. We validated the efficacy of our method on modern DCNN architectures using the MNIST, CIFAR, and ImageNet datasets, achieving competitive performance compared to existing structured pruning approaches.

Electrostatic Force Regularization for Neural Structured Pruning

TL;DR

A novel method is proposed that, for the first time, incorporates the concepts of charge and electrostatic force from physics into the training process of DCNNs, and simultaneously optimizes weights and ranks filter importance, all without the need for extensive fine-tuning.

Abstract

The demand for deploying deep convolutional neural networks (DCNNs) on resource-constrained devices for real-time applications remains substantial. However, existing state-of-the-art structured pruning methods often involve intricate implementations, require modifications to the original network architectures, and necessitate an extensive fine-tuning phase. To overcome these challenges, we propose a novel method that, for the first time, incorporates the concepts of charge and electrostatic force from physics into the training process of DCNNs. The magnitude of this force is directly proportional to the product of the charges of the convolution filter and the source filter, and inversely proportional to the square of the distance between them. We applied this electrostatic-like force to the convolution filters, either attracting filters with opposite charges toward non-zero weights or repelling filters with like charges toward zero weights. Consequently, filters subject to repulsive forces have their weights reduced to zero, enabling their removal, while the attractive forces preserve filters with significant weights that retain information. Unlike conventional methods, our approach is straightforward to implement, does not require any architectural modifications, and simultaneously optimizes weights and ranks filter importance, all without the need for extensive fine-tuning. We validated the efficacy of our method on modern DCNN architectures using the MNIST, CIFAR, and ImageNet datasets, achieving competitive performance compared to existing structured pruning approaches.

Paper Structure

This paper contains 23 sections, 11 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Motivation
  5. Proposed Method
  6. Experimental Setup
  7. Datasets and Networks
  8. Implementation
  9. Electrostatic Force-based Training
  10. Pruning
  11. Fine-Tuning
  12. Results
  13. Pruning Results
  14. Training Overhead
  15. Fine-Tuning Results
  16. ...and 8 more sections

Figures (3)

  • Figure 1: An illustration of electrostatic force-based training. A convolutional layer with ten filters is used as an example. The color shading represents the magnitude of each filter, with the source filter exhibiting the largest magnitude. The signs (plus and minus) indicate the polarity of the filter weights, corresponding to the filter charge (positive or negative). Initially, the convolutional layer contains filters with either random or pretrained weights. Filters with charges similar to the source filter (i.e., positive charges) experience repulsive forces, pushing their weights toward zero. Conversely, filters with opposite charges experience attractive forces, pulling their weights toward non-zero values.
  • Figure 2: Top-1 accuracy of pruned ResNet-56 and VGG-19 models, initialized with pretrained weights and trained with L$_1$-norm and electrostatic force at four distinct electrostatic force rates, without any fine-tuning.
  • Figure 3: Normalized filter L$_1$-norm of layers 1 and 2 for the ResNet-56/CIFAR-10 model trained with and without electrostatic force, and with L$_1$-norm.