Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks

TL;DR

The paper introduces Random Sparse Lifts, a computation-graph framework that constructs large finite neural networks by lifting base perceptron modules into bigger, often sparse architectures. It formalizes a tangent-approximation condition, enabling a PAC-style convergence guarantee for gradient-flow training as the lift size grows, without relying on classical overparameterization assumptions. By defining perceptron modules on DAGs and describing how lifts via graph homomorphisms reproduce standard architectures (including transformer-like blocks), it provides a unified theory connecting architecture geometry, subnetwork coverage, and convergence. The main result shows that, given a witness network achieving low loss, random sparse lifts converge to the target performance with high probability, leveraging tools from algebraic topology and random graph theory to reason about coverage, morphisms, and activations throughout training, ultimately offering a principled path toward deep-learning convergence theory for large, sparse networks.

Abstract

We present a framework to define a large class of neural networks for which, by construction, training by gradient flow provably reaches arbitrarily low loss when the number of parameters grows. Distinct from the fixed-space global optimality of non-convex optimization, this new form of convergence, and the techniques introduced to prove such convergence, pave the way for a usable deep learning convergence theory in the near future, without overparameterization assumptions relating the number of parameters and training samples. We define these architectures from a simple computation graph and a mechanism to lift it, thus increasing the number of parameters, generalizing the idea of increasing the widths of multi-layer perceptrons. We show that architectures similar to most common deep learning models are present in this class, obtained by sparsifying the weight tensors of usual architectures at initialization. Leveraging tools of algebraic topology and random graph theory, we use the computation graph's geometry to propagate properties guaranteeing convergence to any precision for these large sparse models.

Figures (12)

• Figure 1: Tangent approximation to $\varepsilon$ error ($\varepsilon$-ball depicted by a dashed line) for fixed $s \in \mathcal{S}$, $g \in G_s$ with $f = F_{(s,g)}$. If $\lVert f_{\theta^\star} - f^\star \rVert^2_\mathcal{D} < \varepsilon$, then ${\theta^\star} \in \mathcal{A}(0, \varepsilon)$. In contrast, $\theta \in \mathcal{A}(\lVert u \rVert, \varepsilon) \setminus \mathcal{A}(0, \varepsilon)$.
• Figure 2: Illustration of the construction of a pullback bundle, with $\pi : U \to V$ indicated by arrows. The top row is a set $V$ with 3 elements pictured as circles, and bottom row is $U$ with 6 elements. The vector spaces attached to each element are depicted next to each corresponding circle.
• Figure 3: Generic blueprint notation
• Figure 4: MLP representing functions $\mathbb{R}^{\{r,g,b\}} \to \mathbb{R}^4$
• Figure 5: Sparse lift graphs, denoted $G$ in the more general Def. \ref{['def:lift']}, not covered by Def. \ref{['def:fc-lift']}.

Theorems & Definitions (58)

• Theorem 2.1: Probably approximately correct convergence in loss
• Definition 3.1: Euclidean bundle over a finite set
• Definition 3.2: Pullbacks of bundles and sections
• Definition 3.3: Perceptron module
• Definition 3.4: Activation map, or "forward" function, of a perceptron module
• Definition 3.5: Lift of perceptron modules
• Definition 3.6: Fully-connected lift
• Definition 4.1: Perceptron with linear readout
• Definition 4.2: Random sparse lift
• Theorem 4.3: Probable approximate correctness of random sparse lifts under gradient flow