Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generation

TL;DR

A matricidal free energy maximizing autoencoder that reliably produces Gaussian codes is proposed and its application to underdetermined inverse problems is shown.

Abstract

We introduce a novel regularization scheme for autoencoders based on matricial free energy. Our approach defines a differentiable loss function in terms of the singular values of the code matrix (code dimension x batch size). From the standpoint of free probability an d random matrix theory, this loss achieves its minimum when the singular value distribution of the code matrix coincides with that of an appropriately sculpted random metric with i.i.d. Gaussian entries. Empirical simulations demonstrate that minimizing the negative matricial free energy through standard stochastic gradient-based training yields Gaussian-like codes that generalize across training and test sets. Building on this foundation, we propose a matricidal free energy maximizing autoencoder that reliably produces Gaussian codes and show its application to underdetermined inverse problems.

Figures (17)

• Figure 1: (a) Scatter plot of the training dataset generated as described in Section \ref{['sec:free encoder']}. (b) The training loss and test curves for a batch of size 256 data points embedded in 32-dimensional space. The blue line is the training loss for a random batch, the orange line is for a random test batch, and the red line is the theoretical minimum computed by sampling a matrix with i.i.d. $\mathcal{N}(0,1)$ entries.
• Figure 2: Visualization of the outputs of a free Gaussianizing encoder trained as described in Section \ref{['sec:free encoder']}. The top two rows show the histogram of the entries of entries of $\mathcal{E}_{\mathrm{opt}}(X_b)$, a Q-Q (or quantile-quantile) plot for the entries of $\mathcal{E}_{\mathrm{opt}}(X_b)$ which compares the quantiles of the empirical data against the quantiles of a theoretical standard normal distribution, and the empirical eigenvalue disribution of $(1/b),\mathcal{E}_{\mathrm{opt}}(X_b)\mathcal{E}_{\mathrm{opt}}(X_b)^\top$ for the training and test dataset relative to the Marčenko-Pastur distribution in (\ref{['eq:mandp']}) with parameter $c = d/b = 32/256$. Row 3 shows the entrywise variance, fourth, and sixth moments (red line is true moments for $\mathcal{N}(0,1)$ target) of the training and test data as the free Gaussianizing encoder is trained.
• Figure 3: Figure showing the Free Loss vs MSE for training an autoencoder for different values of $\tau$ as described in Section \ref{['sec:free autoencoder']}. Here the theoretical minimum Free Loss depicted at $-34.69$ is the value obtained by averaging the empirical Free Loss of many independent $32 \times 256$ i.i.d. Gaussian matrices.
• Figure 4: Histograms of autoencoder code entries under three training regimes as described in Section \ref{['sec:free autoencoder']}: Tikhonov regularization ($\tau=1$), no regularization ($\tau=0$), and free-loss regularization ($\tau=1$), evaluated on the test set after 2000 epochs.
• Figure 5: Histogram of the recovered coordinate conditioned on the sign of the projected coordinate $z$ for the free regularized autoencoder (red) versus the Tikhonov regularized (classical) autoencoder (blue) for the setting where $z$ in (\ref{['eq:recover-free']}) is positive. The ground truth is plotted in blue. We used $\rho = 0.0005$ and performed 5000 gradient steps using SGD with a learning rate of $10^{-3}$ to optimize \ref{['eq:recover-free']} for $x$ in (\ref{['eq:recover-free']}).

Theorems & Definitions (11)

• Definition 2.1: Empirical Spectral Distribution (ESD)
• Definition 2.2: Marčenko-Pastur distribution
• Proposition 2.3
• proof
• Definition 2.4: Voiculescu Free Entropy
• Proposition 2.5: Maximization Principle for the Marčenko-Pastur distribution
• proof
• Remark 3.1: Training Length
• Proposition A.1: Maximization Principle for Free Poisson Distribution
• Proposition A.2: Maximization Principle for Marčenko-Pastur Distribution