What Regularized Auto-Encoders Learn from the Data Generating Distribution

TL;DR

This work reveals that regularized auto-encoders, including denoising and contractive variants, implicitly learn local properties of the data-generating density. By analyzing the optimal reconstruction under a regularized loss, it shows that the reconstruction drift encodes the score, and its Jacobian encodes the Hessian of the log-density, enabling estimation of local mean and facilitating sampling via Metropolis-Hastings using energy-difference approximations. The authors establish strong, parametric-agnostic connections to score matching, provide empirical demonstrations on toy manifolds, and propose practical sampling methods that bypass partition-function calculations. Collectively, the results offer a principled interpretation of reconstruction error and present a scalable alternative to likelihood-based density estimation for unsupervised learning and generative sampling.

Abstract

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

Figures (7)

• Figure 1: Regularization forces the auto-encoder to become less sensitive to the input, but minimizing reconstruction error forces it to remain sensitive to variations along the manifold of high density. Hence the representation and reconstruction end up capturing well variations on the manifold while mostly ignoring variations orthogonal to it.
• Figure 2: The reconstruction function $r(x)$ (in turquoise) which would be learned by a high-capacity auto-encoder on a 1-dimensional input, i.e., minimizing reconstruction error at the training examples$x_i$ (with $r(x_i)$ in red) while trying to be as constant as possible otherwise. The figure is used to exagerate and illustrate the effect of the regularizer (corresponding to a large $\sigma^2$ in the loss function $\mathcal{L}$ later described by (\ref{['eqn:RCAE_reconstruction_loss']})). The dotted line is the identity reconstruction (which might be obtained without the regularizer). The blue arrows shows the vector field of $r(x)-x$ pointing towards high density peaks as estimated by the model, and estimating the score (log-density derivative), as shown in this paper.
• Figure 3: the density $p(x)$ and its score for a simple one-dimensional example.
• Figure 4: Comparing the approximation of the score of $p$ given by discrete versions of optimally trained auto-encoders with infinite capacity. The approximations given by the RCAE are in orange while the approximations given by the DAE are in purple. The results are shown for decreasing values of $\sigma \in \{1.00, 0.31, 0.16, 0.06\}$ that have been selected for their visual appeal. As expected, we see in that the RCAE (orange) and DAE (purple) approximations of the score are close to each other as predicted by Proposition \ref{['prp:DAE-connection-RCAE']}. Moreover, they are also converging to the true score (green) as predicted by Theorem \ref{['thm:DAE-optimal-solution']} and Theorem \ref{['thm:calcvarminloss']}.
• Figure 5: The original 2-D data from the data generating density $p(x)$ is plotted along with the vector field defined by the values of $r(x)-x$ for trained auto-encoders (corresponding to the estimation of the score $\frac{\partial \log p(x)}{\partial x}$).

Theorems & Definitions (14)

• Theorem 1
• Proposition 1
• Theorem 2
• Theorem 1
• Proposition 1
• Theorem 2
• Theorem 3
• Proposition 4
• Theorem 5
• Theorem 6