Deep Neural Networks as Iterated Function Systems and a Generalization Bound

TL;DR

The paper reframes deep neural networks as stochastic iterated function systems, unifying depth dynamics with random dynamical systems to address stability and generalization in generative modeling. By mapping ResNet, Transformer, and MoE architectures to place-dependent IFSs, it proves existence and uniqueness of invariant measures under an average-contractivity condition and derives a Wasserstein-based generalization bound via a collage error. This bound motivates a practical training objective that minimizes the discrepancy between the empirical distribution and its image under the learned transfer operator, demonstrated on simple 2D illustrations and latent representations of MNIST, CelebA, and CIFAR-10. The results highlight an expressivity-stability trade-off, show meaningful attractors under contraction, and point to future directions for relaxing contraction requirements and extending collage-based methods to higher-dimensional data spaces.

Abstract

Deep neural networks (DNNs) achieve remarkable performance on a wide range of tasks, yet their mathematical analysis remains fragmented: stability and generalization are typically studied in disparate frameworks and on a case-by-case basis. Architecturally, DNNs rely on the recursive application of parametrized functions, a mechanism that can be unstable and difficult to train, making stability a primary concern. Even when training succeeds, there are few rigorous results on how well such models generalize beyond the observed data, especially in the generative setting. In this work, we leverage the theory of stochastic Iterated Function Systems (IFS) and show that two important deep architectures can be viewed as, or canonically associated with, place-dependent IFS. This connection allows us to import results from random dynamical systems to (i) establish the existence and uniqueness of invariant measures under suitable contractivity assumptions, and (ii) derive a Wasserstein generalization bound for generative modeling. The bound naturally leads to a new training objective that directly controls the collage-type approximation error between the data distribution and its image under the learned transfer operator. We illustrate the theory on a controlled 2D example and empirically evaluate the proposed objective on standard image datasets (MNIST, CelebA, CIFAR-10).

Figures (8)

• Figure 1: Schematic illustration of fractal generation with IFSs. Top: classical fractal attractor construction. Bottom: fractal encoding of an image, where contractive maps learned from patch correspondences define an IFS whose invariant set approximates the target image.
• Figure 2: Left: Training collage error $\varepsilon(n,\theta)$ over epochs. Center: Training data and samples of the attractor measure of the trained MoE IFS. Right: Estimated generalization error and its bound (Theorem \ref{['thm:generalization-bound']}) for different contraction constants.
• Figure 3: From left to right : attractors obtained with ResNet with zero bias, ResNet with learned bias and Transformer architectures trained with collage error minimization.
• Figure 4: Samples from the learned MoE IFS on latent MNIST, CelebA, and CIFAR-10 representations (left to right).
• Figure 5: Training data and samples of the attractor measure of the trained IFS for contraction constants $c \in \{0.3,0.5,0.7\}$ (left to right).

Theorems & Definitions (39)

• Corollary 1: "Collage" in Metric Space
• Definition 1: IFS
• Theorem 1: Original Collage Theorem
• Definition 2: Stochastic IFS
• Definition 3: Markov operator, transfer operator and invariant measure
• Theorem 2: Invariant measure for P-IFS
• Proposition 1: Support of invariant measure and Hutchinson fixed-point
• Theorem 3: Collage Theorem on measure
• Definition 4: ReLU residual block
• Definition 5: Residual branch family and selector kernel