Why should autoencoders work?

TL;DR

This work explains why autoencoders can effectively compress data lying on low-dimensional manifolds within high-dimensional spaces despite topological obstacles. It proves a PAC-style theorem: for $K\subseteq\mathbb{R}^n$ being a finite union of smooth $k$-dimensional manifolds with boundary, there exists a small exceptional set $K_0$ (in intrinsic measure) such that, for any $\varepsilon>0$, encoder/decoder pairs from universal-approximation classes can achieve $\|G(F(x)) - x\| < \varepsilon$ on $K\setminus K_0$, providing a theoretical justification for practical success. The results combine differential-topology constructs (intrinsic measures, cut loci, reach) with universal approximation to yield a constructive existence claim, complemented by a lower-bound showing global reconstruction is limited by geometry. A numerical example with two interlaced circles demonstrates the method’s ability to open topological obstructions in practice, while Section 4 establishes that global perfection is impossible in general due to reach/topology constraints. The discussion points to rich future directions, including time-series dynamics, Koopman embeddings, and stratified-set extensions, broadening the impact of these theoretical insights on manifold learning and representation learning.

Abstract

Deep neural network autoencoders are routinely used computationally for model reduction. They allow recognizing the intrinsic dimension of data that lie in a $k$-dimensional subset $K$ of an input Euclidean space $\mathbb{R}^n$. The underlying idea is to obtain both an encoding layer that maps $\mathbb{R}^n$ into $\mathbb{R}^k$ (called the bottleneck layer or the space of latent variables) and a decoding layer that maps $\mathbb{R}^k$ back into $\mathbb{R}^n$, in such a way that the input data from the set $K$ is recovered when composing the two maps. This is achieved by adjusting parameters (weights) in the network to minimize the discrepancy between the input and the reconstructed output. Since neural networks (with continuous activation functions) compute continuous maps, the existence of a network that achieves perfect reconstruction would imply that $K$ is homeomorphic to a $k$-dimensional subset of $\mathbb{R}^k$, so clearly there are topological obstructions to finding such a network. On the other hand, in practice the technique is found to "work" well, which leads one to ask if there is a way to explain this effectiveness. We show that, up to small errors, indeed the method is guaranteed to work. This is done by appealing to certain facts from differential topology. A computational example is also included to illustrate the ideas.

Figures (9)

• Figure 1: An autoencoder consists of an encoding layer, which maps inputs that lie in a subset $K$ of $\mathbb{R}^n$ ($n=12$ in this illustration) into a hidden or latent layer of points in $\mathbb{R}^k$ (here $k=4$), followed by a decoding layer mapping $\mathbb{R}^k$ back into $\mathbb{R}^n$. The goal is to make the decoded vectors (in red) match the data vectors (in blue). In a perfect autoencoder, $G(F(x))=x$ for all $x$ in $K$. Due to topological obstructions, a more realistic goal is to achieve $G(F(x))\approx x$ for all $x$ in a large subset of $K$.
• Figure 2: Left: Two interlaced unit circles, one centered at $x=y=0$ in the plane $z=0$ (blue), and another centered at $x=1$, $z=0$ in the plane $y=0$ (red). The circles are parameterized as $x(\theta) = (\cos(\theta),\sin(\theta),0)$ and $x(\theta) = (1+\cos(\theta),0,\sin(\theta))$ respectively, with $\theta\in[0,2\pi]$. Right: The output of the autoencoder for the two interlaced unit circles, one centered at $x=y=0$ in the plane $z=0$ (blue), and another centered at $x=1$, $z=0$ in the plane $y=0$ (red). The network learning algorithm automatically picked the points at which the circles should be "opened up" to avoid the topological obstruction.
• Figure 3: The architecture used in the computational example. For clarity in the illustration, only 6 units are depicted in each layer of the encoder and decoder, but the number used was 128.
• Figure 4: The errors $\lVert G(F(x))-x\rVert$ on the two cirles. The $x$-axis shows the index $k$ representing the $k$th point in the respective circle, where $\theta = 2\pi k/1000$.
• Figure 5: Left: The bottleneck layer, showing the images of the blue and red circles. Middle and Right: The encoding maps for the two circles. The $x$-axis is the angle $\theta$ in a $2\pi$ parametrization of the unit circles. The $y$-axis is the coordinate in the one-dimensional bottleneck layer.

Theorems & Definitions (27)

• Theorem 1
• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Lemma 1
• Remark 5
• proof
• Lemma 2
• proof