Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularization

Yehonathan Refael, Iftach Arbel, Wasim Huleihel

TL;DR

This paper tackles the challenge of deploying large DNNs on resource-limited devices by introducing WGSEF, a convex relaxation that generalizes the Sparse Envelop Function to structured, group-level sparsity. By selecting up to k predefined groups, WGSEF enables hardware-aware pruning with a proximal operator that scales linearly with the number of parameters, and a proximal-gradient training scheme that can predefine sparsity levels without sacrificing accuracy. The approach is validated across multiple architectures and datasets, showing higher group sparsity, competitive or improved accuracy, and substantial speedups and FLOPs reductions compared to existing pruning methods. These results highlight WGSEF as a practical, hardware-aligned tool for neural network compression and acceleration with broad applicability to structured sparsity settings.

Abstract

The extensive need for computational resources poses a significant obstacle to deploying large-scale Deep Neural Networks (DNN) on devices with constrained resources. At the same time, studies have demonstrated that a significant number of these DNN parameters are redundant and extraneous. In this paper, we introduce a novel approach for learning structured sparse neural networks, aimed at bridging the DNN hardware deployment challenges. We develop a novel regularization technique, termed Weighted Group Sparse Envelope Function (WGSEF), generalizing the Sparse Envelop Function (SEF), to select (or nullify) neuron groups, thereby reducing redundancy and enhancing computational efficiency. The method speeds up inference time and aims to reduce memory demand and power consumption, thanks to its adaptability which lets any hardware specify group definitions, such as filters, channels, filter shapes, layer depths, a single parameter (unstructured), etc. The properties of the WGSEF enable the pre-definition of a desired sparsity level to be achieved at the training convergence. In the case of redundant parameters, this approach maintains negligible network accuracy degradation or can even lead to improvements in accuracy. Our method efficiently computes the WGSEF regularizer and its proximal operator, in a worst-case linear complexity relative to the number of group variables. Employing a proximal-gradient-based optimization technique, to train the model, it tackles the non-convex minimization problem incorporating the neural network loss and the WGSEF. Finally, we experiment and illustrate the efficiency of our proposed method in terms of the compression ratio, accuracy, and inference latency.

Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularization

TL;DR

This paper tackles the challenge of deploying large DNNs on resource-limited devices by introducing WGSEF, a convex relaxation that generalizes the Sparse Envelop Function to structured, group-level sparsity. By selecting up to k predefined groups, WGSEF enables hardware-aware pruning with a proximal operator that scales linearly with the number of parameters, and a proximal-gradient training scheme that can predefine sparsity levels without sacrificing accuracy. The approach is validated across multiple architectures and datasets, showing higher group sparsity, competitive or improved accuracy, and substantial speedups and FLOPs reductions compared to existing pruning methods. These results highlight WGSEF as a practical, hardware-aligned tool for neural network compression and acceleration with broad applicability to structured sparsity settings.

Abstract

The extensive need for computational resources poses a significant obstacle to deploying large-scale Deep Neural Networks (DNN) on devices with constrained resources. At the same time, studies have demonstrated that a significant number of these DNN parameters are redundant and extraneous. In this paper, we introduce a novel approach for learning structured sparse neural networks, aimed at bridging the DNN hardware deployment challenges. We develop a novel regularization technique, termed Weighted Group Sparse Envelope Function (WGSEF), generalizing the Sparse Envelop Function (SEF), to select (or nullify) neuron groups, thereby reducing redundancy and enhancing computational efficiency. The method speeds up inference time and aims to reduce memory demand and power consumption, thanks to its adaptability which lets any hardware specify group definitions, such as filters, channels, filter shapes, layer depths, a single parameter (unstructured), etc. The properties of the WGSEF enable the pre-definition of a desired sparsity level to be achieved at the training convergence. In the case of redundant parameters, this approach maintains negligible network accuracy degradation or can even lead to improvements in accuracy. Our method efficiently computes the WGSEF regularizer and its proximal operator, in a worst-case linear complexity relative to the number of group variables. Employing a proximal-gradient-based optimization technique, to train the model, it tackles the non-convex minimization problem incorporating the neural network loss and the WGSEF. Finally, we experiment and illustrate the efficiency of our proposed method in terms of the compression ratio, accuracy, and inference latency.
Paper Structure (28 sections, 6 theorems, 47 equations, 2 figures, 6 tables, 3 algorithms)

This paper contains 28 sections, 6 theorems, 47 equations, 2 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work.
  3. Notation.
  4. Structured sparsity via WGSEF
  5. Problem formulation
  6. Convex relaxation
  7. Proximal mapping of the WGSEF
  8. Optimization procedure
  9. Experiments
  10. Evaluation of sparsity-inducing optimization methods
  11. Evaluation of different sparsity-inducing regularizers
  12. Evaluation of different state-of-the-art pruning techniques on ImageNet
  13. Evaluation of different group structures
  14. Impact of sparsification levels on model error and training dynamics
  15. Discussion
  16. ...and 13 more sections

Key Result

Lemma 1

Let subsets $s_1, s_2, \ldots ,s_m$ be a set of $m\leq n$ disjoint indexes that partition $[n]$. Then, for any $\tilde{\theta}\in \mathbb{R}^n$,

Figures (2)

  • Figure 1: Ratio of the sparse filters in the convolutional layers, according to the order of the layers in the resnet18 model, as obtained by Algorithm \ref{['prox_sgd_gse']}, and corresponds to the experiment in row 3 of table \ref{['Table1']}.
  • Figure 2: The graph on the left shows the level of sparseness as a function of epoch number, while the graph on the right shows the model's accuracy as a function of epoch number.

Theorems & Definitions (19)

  • Definition 1: Group projection
  • Example 1
  • Remark 1: Generalization of SEF
  • Lemma 1: The $k$ weighted group sparse envelop conjugate
  • Lemma 2: The variational bi-conjugate $k$ weighted group sparse envelop
  • Corollary 2.1
  • Remark 2
  • Lemma 3
  • Corollary 3.1: The proximal operator of ${\mathcal{GS}}_k$
  • Remark 3: Root search application for function \ref{['prox_root_search']}
  • ...and 9 more