Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Digital implementations of deep feature extractors are intrinsically informative

Max Getter

TL;DR

The paper investigates how much information digital deep feature extractors retain by bounding the speed of energy propagation across layers. It introduces a unified framework for general measure spaces and specialized results for scattering CNNs on LCA groups, deriving energy-decay bounds that are often exponential. The main results include a generic bound $W_N(f) \le (||f||^2 - ||A^{(0)}f||^2) \cdot \prod_{\ell=1}^{N-1} (1 - C^{(\ell)})$ and setting-specific rates, such as $W_N(f) \le (||f||^2 - ||f * χ||^2) \cdot (1 - 1/S)^{N-1}$ for compact G with bounded support, or $W_N(f) \le (||f||^2 - ||f * χ||^2) \cdot \alpha(Ψ,Γ)^{N-1}$ for general LCA groups. These results justify that digital feature extractors are intrinsically informative and quantify how energy concentrates across depth, guiding design choices for both theory and practice. The work connects energy propagation to spectral and structural properties (e.g., frame bounds, measure space minima, and Fourier supports), offering principled insight into the stability and expressiveness of deep representations in diverse domains.

Abstract

Rapid information (energy) propagation in deep feature extractors is crucial to balance computational complexity versus expressiveness as a representation of the input. We prove an upper bound for the speed of energy propagation in a unified framework that covers different neural network models, both over Euclidean and non-Euclidean domains. Additional structural information about the signal domain can be used to explicitly determine or improve the rate of decay. To illustrate this, we show global exponential energy decay for a range of 1) feature extractors with discrete-domain input signals, and 2) convolutional neural networks (CNNs) via scattering over locally compact abelian (LCA) groups.

Digital implementations of deep feature extractors are intrinsically informative

TL;DR

The paper investigates how much information digital deep feature extractors retain by bounding the speed of energy propagation across layers. It introduces a unified framework for general measure spaces and specialized results for scattering CNNs on LCA groups, deriving energy-decay bounds that are often exponential. The main results include a generic bound and setting-specific rates, such as for compact G with bounded support, or for general LCA groups. These results justify that digital feature extractors are intrinsically informative and quantify how energy concentrates across depth, guiding design choices for both theory and practice. The work connects energy propagation to spectral and structural properties (e.g., frame bounds, measure space minima, and Fourier supports), offering principled insight into the stability and expressiveness of deep representations in diverse domains.

Abstract

Rapid information (energy) propagation in deep feature extractors is crucial to balance computational complexity versus expressiveness as a representation of the input. We prove an upper bound for the speed of energy propagation in a unified framework that covers different neural network models, both over Euclidean and non-Euclidean domains. Additional structural information about the signal domain can be used to explicitly determine or improve the rate of decay. To illustrate this, we show global exponential energy decay for a range of 1) feature extractors with discrete-domain input signals, and 2) convolutional neural networks (CNNs) via scattering over locally compact abelian (LCA) groups.

Paper Structure

This paper contains 15 sections, 5 theorems, 38 equations.

Table of Contents

  1. Introduction
  2. Prior work
  3. Our contributions
  4. Information retention in general feature extractors
  5. A general model for feature extraction
  6. Energy conservation
  7. A generic rate for energy decay
  8. Information retention in scattering CNNs over LCA groups
  9. LCA group scattering
  10. A setting-specific rate for energy decay
  11. Appendix
  12. Proof of Theorem \ref{['thm: energy decay in the general setting']}
  13. Proof of Corollary \ref{['cor: generic rate for discrete measure spaces']}
  14. Proof of Theorem \ref{['thm: rate for finite LCA group scattering']}
  15. Proof of Theorem \ref{['thm: rate for general LCA group scattering']}

Key Result

Proposition 2.2

We have, for every $f\in \mathcal{H}$ and every $N\in \mathbb{N}$, where $W_N(f):=\sum_{p\in \mathcal{P}^{(N)}} \left\lVert U[p]f\right\rVert^2$. Thus, Further, $S:\mathcal{H} \to \bigoplus_{\ell=0}^\infty \bigoplus_{p\in \mathcal{P}^{(\ell)}} \mathcal{H}^{(\ell)}$ is nonexpansive with $S(0)=0$, hence norm-decreasing.

Theorems & Definitions (15)

  • Proposition 2.2
  • Remark 2.3
  • Theorem 2.4
  • proof
  • Remark 2.5
  • Corollary 2.6
  • proof
  • Remark 2.7
  • Theorem 3.2
  • proof
  • ...and 5 more