Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks

TL;DR

The paper develops a comprehensive theory that connects deep sparse coding (DSC) with convolutional neural networks (CNNs), proving uniqueness and stability of multi-layer sparse representations and establishing CNN-based convergence guarantees for sparse feature extraction. It introduces two CNN-based strategies to solve DSC problems and extends the analysis to ReLU-type and generalized activations as well as architectures like self-attention/transformers. Empirically, sparsity-promoting training (via ll_1 penalties) improves performance on image Classification and Segmentation tasks while yielding sparser internal representations. The work highlights the broad applicability of sparse-coding principles across modern deep architectures and provides practical guidance for designing efficient, interpretable networks.

Abstract

In this work, we explore the intersection of sparse coding theory and deep learning to enhance our understanding of feature extraction capabilities in advanced neural network architectures. We begin by introducing a novel class of Deep Sparse Coding (DSC) models and establish a thorough theoretical analysis of their uniqueness and stability properties. By applying iterative algorithms to these DSC models, we derive convergence rates for convolutional neural networks (CNNs) in their ability to extract sparse features. This provides a strong theoretical foundation for the use of CNNs in sparse feature-learning tasks. We additionally extend this convergence analysis to more general neural network architectures, including those with diverse activation functions, as well as self-attention and transformer-based models. This broadens the applicability of our findings to a wide range of deep learning methods for the extraction of deep-sparse features. Inspired by the strong connection between sparse coding and CNNs, we also explore training strategies to encourage neural networks to learn sparser features. Through numerical experiments, we demonstrate the effectiveness of these approaches, providing valuable insight for the design of efficient and interpretable deep learning models.

Figures (7)

• Figure 1: Test accuracy over CIFAR10.
• Figure 2: Training loss over CIFAR10.
• Figure 3: Training loss of VGG11 over first $50$ epoch.
• Figure 4: Training loss and $\ell_1$ penalty ($\sum_j \omega_j \|\mathbf x_j\|_1$) during training.
• Figure 5: Average sparsity of features produced by Unet over DUT-OMRON.

Theorems & Definitions (39)

• Definition 1: ML-CSC problem
• Definition 2: Deep sparse coding problem
• Definition 3
• Theorem 1: Uniqueness via mutual coherence without noise
• proof
• Definition 4: $DSC_{1}$
• Theorem 2: Coincidence between $\ell_0$ and $\ell_1$
• proof
• Theorem 3: Stability of $(DSC_{0,\bm \lambda}^{\bm\varepsilon})$ problem
• proof