Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-Linear Time Projection onto the $\ell_{1,\infty}$ Ball; Application to Sparse Autoencoders

Guillaume Perez, Laurent Condat, Michel Barlaud

TL;DR

This work tackles the efficient sparsity-inducing projection onto the $\ell_{1,\infty}$ ball for neural networks. It introduces a near-linear time projection algorithm with worst-case complexity $\mathcal{O}(nm+J\log(nm))$, where $J$ vanishes as sparsity grows, and proves exact finite-time convergence. The method leverages a global threshold $\theta$ and per-column updates, with a heap-based implementation that dramatically speeds up sparse projections. The authors apply the projection to a supervised autoencoder to achieve strong feature selection in biology, achieving high sparsity with improved or maintained accuracy and showcasing significant speedups over existing approaches. Overall, the approach enables scalable, interpretable structured sparsity for large-scale neural networks and high-dimensional biological data.

Abstract

Looking for sparsity is nowadays crucial to speed up the training of large-scale neural networks. Projections onto the $\ell_{1,2}$ and $\ell_{1,\infty}$ are among the most efficient techniques to sparsify and reduce the overall cost of neural networks. In this paper, we introduce a new projection algorithm for the $\ell_{1,\infty}$ norm ball. The worst-case time complexity of this algorithm is $\mathcal{O}\big(nm+J\log(nm)\big)$ for a matrix in $\mathbb{R}^{n\times m}$. $J$ is a term that tends to 0 when the sparsity is high, and to $nm$ when the sparsity is low. Its implementation is easy and it is guaranteed to converge to the exact solution in a finite time. Moreover, we propose to incorporate the $\ell_{1,\infty}$ ball projection while training an autoencoder to enforce feature selection and sparsity of the weights. Sparsification appears in the encoder to primarily do feature selection due to our application in biology, where only a very small part ($<2\%$) of the data is relevant. We show that both in the biological case and in the general case of sparsity that our method is the fastest.

Near-Linear Time Projection onto the $\ell_{1,\infty}$ Ball; Application to Sparse Autoencoders

TL;DR

This work tackles the efficient sparsity-inducing projection onto the ball for neural networks. It introduces a near-linear time projection algorithm with worst-case complexity , where vanishes as sparsity grows, and proves exact finite-time convergence. The method leverages a global threshold and per-column updates, with a heap-based implementation that dramatically speeds up sparse projections. The authors apply the projection to a supervised autoencoder to achieve strong feature selection in biology, achieving high sparsity with improved or maintained accuracy and showcasing significant speedups over existing approaches. Overall, the approach enables scalable, interpretable structured sparsity for large-scale neural networks and high-dimensional biological data.

Abstract

Looking for sparsity is nowadays crucial to speed up the training of large-scale neural networks. Projections onto the and are among the most efficient techniques to sparsify and reduce the overall cost of neural networks. In this paper, we introduce a new projection algorithm for the norm ball. The worst-case time complexity of this algorithm is for a matrix in . is a term that tends to 0 when the sparsity is high, and to when the sparsity is low. Its implementation is easy and it is guaranteed to converge to the exact solution in a finite time. Moreover, we propose to incorporate the ball projection while training an autoencoder to enforce feature selection and sparsity of the weights. Sparsification appears in the encoder to primarily do feature selection due to our application in biology, where only a very small part () of the data is relevant. We show that both in the biological case and in the general case of sparsity that our method is the fastest.
Paper Structure (17 sections, 5 theorems, 23 equations, 6 figures, 1 table, 2 algorithms)

This paper contains 17 sections, 5 theorems, 23 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. $\ell_{1,\infty}$ ball, simplex, and Projection
  3. Definitions
  4. Properties
  5. Relation between the $\ell_{1,\infty}$ and $\ell_{\infty,1}$ norms
  6. Projection algorithms
  7. Algorithmic mechanisms
  8. Total order
  9. Build $P'$ then find $\theta$ quattoni2009efficient
  10. Proposed Projection Algorithm
  11. Implementation
  12. columns eliminations
  13. Projection Experiments
  14. Supervised Autoencoder (SAE) framework
  15. SAE experimental results
  16. ...and 2 more sections

Key Result

Lemma 1

At the optimal solution of problem eq:prob:P--eq:P1inf:lasteq, there exists a constant $\theta \geq 0$ such that for every $j=1,\ldots,m$: either $\mu_j > 0$ and $\sum_i(Y_{i,j}-X_{i,j})=\theta$; or $\mu_j = 0$, $\sum_i Y_{i,j} \leq \theta$, and $\forall i=1,\ldots,n$, $X_{i,j}=0$.

Figures (6)

  • Figure 1: Impact of the radius on the sparsity of the matrix. Comparison of the projection times.
  • Figure 2: Projection time for matrix sizes (left) 1000x10000, (right) 10000x1000.
  • Figure 3: (left) Projection time for a fixed $n$. (right) Projection time for a fixed $m$
  • Figure 4: $J$ complexity constant value as a function of the resulting sparsity. The % is of the global size of the matrix ($nm$).
  • Figure 5: Synthetic data. Left: sparsity and parameter $\theta$ as a function of the radius C.Right:Parameter $\theta$ as a function of the radius C.
  • ...and 1 more figures

Theorems & Definitions (6)

  • Lemma 1
  • Proposition 1
  • proof
  • Proposition 2
  • Proposition 3
  • Lemma 2