Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networks

Guillaume Perez, Michel Barlaud

TL;DR

The paper tackles the high computational cost of projecting onto the structured $\ell_{1,\infty}$ ball, which impedes scalable sparsity in neural networks. It introduces bi-level and multi-level projection frameworks that decompose the projection into independent, parallelizable steps, achieving $O(nm)$ time for matrices and $O(n+m)$ under full parallelism, with exponential speedups when extended to tensors. The approach extends to related norms ($\ell_{1,1}$, $\ell_{1,2}$) and provides tri- and multi-level tensor projections, along with a comprehensive implementation and experimental validation on synthetic and biomedical data, including supervised autoencoders. Experimental results show the bi-level method is at least 2.5x faster than the current fastest algorithms while maintaining accuracy and improving sparsity, highlighting its practical impact for efficient structured sparsity in large-scale neural networks.

Abstract

The $\ell_{1,\infty}$ norm is an efficient structured projection but the complexity of the best algorithm is unfortunately $\mathcal{O}\big(n m \log(n m)\big)$ for a matrix in $\mathbb{R}^{n\times m}$. In this paper, we propose a new bi-level projection method for which we show that the time complexity for the $\ell_{1,\infty}$ norm is only $\mathcal{O}\big(n m \big)$ for a matrix in $\mathbb{R}^{n\times m}$, and $\mathcal{O}\big(n + m \big)$ with full parallel power. We generalize our method to tensors and we propose a new multi-level projection, having an induced decomposition that yields a linear parallel speedup up to an exponential speedup factor, resulting in a time complexity lower-bounded by the sum of the dimensions, instead of the product of the dimensions. we provide a large base of implementation of our framework for bi-level and tri-level (matrices and tensors) for various norms and provides also the parallel implementation. Experiments show that our projection is $2$ times faster than the actual fastest Euclidean algorithms while providing same accuracy and better sparsity in neural networks applications.

Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networks

TL;DR

The paper tackles the high computational cost of projecting onto the structured ball, which impedes scalable sparsity in neural networks. It introduces bi-level and multi-level projection frameworks that decompose the projection into independent, parallelizable steps, achieving time for matrices and under full parallelism, with exponential speedups when extended to tensors. The approach extends to related norms (, ) and provides tri- and multi-level tensor projections, along with a comprehensive implementation and experimental validation on synthetic and biomedical data, including supervised autoencoders. Experimental results show the bi-level method is at least 2.5x faster than the current fastest algorithms while maintaining accuracy and improving sparsity, highlighting its practical impact for efficient structured sparsity in large-scale neural networks.

Abstract

The norm is an efficient structured projection but the complexity of the best algorithm is unfortunately for a matrix in . In this paper, we propose a new bi-level projection method for which we show that the time complexity for the norm is only for a matrix in , and with full parallel power. We generalize our method to tensors and we propose a new multi-level projection, having an induced decomposition that yields a linear parallel speedup up to an exponential speedup factor, resulting in a time complexity lower-bounded by the sum of the dimensions, instead of the product of the dimensions. we provide a large base of implementation of our framework for bi-level and tri-level (matrices and tensors) for various norms and provides also the parallel implementation. Experiments show that our projection is times faster than the actual fastest Euclidean algorithms while providing same accuracy and better sparsity in neural networks applications.
Paper Structure (23 sections, 2 theorems, 18 equations, 5 figures, 3 tables, 8 algorithms)

This paper contains 23 sections, 2 theorems, 18 equations, 5 figures, 3 tables, 8 algorithms.

Table of Contents

  1. Introduction
  2. Motivations and related methods
  3. Contributions
  4. Definitions and Notations
  5. A new bi-level projection formulation
  6. Bi-level $\ell_{1,\infty}$ projection
  7. A new bi-level projection
  8. Computational complexity
  9. Extension to other norms
  10. Extension to the $\ell_{1,1}$ bi-level projection
  11. Extension to bi-level $\ell_{1,2}$ projection
  12. Tensor Generalization
  13. Tri-level Projection Algorithms
  14. Multi-level Projection Algorithms
  15. Experimental results
  16. ...and 8 more sections

Key Result

Proposition 5.3

The multi-level projection is a generalization of the usual projection.

Figures (5)

  • Figure 1: Comparison of our bi-level method with Chu et .al.. Processing time as a function of the radius (matrice fixed 1000x10000)
  • Figure 2: Processing time using C++: our bi-level projection method versus Chu et. al. method. m=1000 and $\eta=1$ fixed .
  • Figure 3: Processing time using C++: our Tri-level projection method d= 32 and n=1000 fixed.
  • Figure 4: Gain factor of the native parallel workload decomposition for various matrix sizes.
  • Figure 5: Accuracy as a function of the radius parameter $\eta$; Left Synthetic data:, Right Lung dataset

Theorems & Definitions (4)

  • Definition 5.1
  • Definition 5.2
  • Proposition 5.3
  • Proposition 5.4