Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rank Reduction Autoencoders

Jad Mounayer, Sebastian Rodriguez, Chady Ghnatios, Charbel Farhat, Francisco Chinesta

TL;DR

Rank Reduction Autoencoders (RRAEs) regularize latent spaces by integrating a truncated SVD into the forward pass, making the bottleneck a function of the latent-space rank rather than a fixed dimension. The adaptive variant aRRAEs automatically discovers the optimal bottleneck during training by pruning the least important singular value and redistributing its influence, without adding extra learning hyperparameters. Empirical results on synthetic data, MNIST, Fashion-MNIST, and CelebA show that RRAEs and aRRAEs yield stable training, scalable bottlenecks, and latent spaces conducive to interpolation and realistic sample generation, outperforming vanilla and other regularizing autoencoders. The work demonstrates that a rank-based bottleneck can provide more meaningful representations and simpler deployment, with practical impact on unsupervised representation learning and generative tasks, especially where data manifolds are complex or high-dimensional.

Abstract

The choice of an appropriate bottleneck dimension and the application of effective regularization are both essential for Autoencoders to learn meaningful representations from unlabeled data. In this paper, we introduce a new class of deterministic autoencoders, Rank Reduction Autoencoders (RRAEs), which regularize their latent spaces by employing a truncated singular value decomposition (SVD) during training. In RRAEs, the bottleneck is defined by the rank of the latent matrix, thereby alleviating the dependence of the encoder/decoder architecture on the bottleneck size. This approach enabled us to propose an adaptive algorithm (aRRAEs) that efficiently determines the optimal bottleneck size during training. We empirically demonstrate that both RRAEs and aRRAEs are stable, scalable, and reliable, as they do not introduce any additional training hyperparameters. We evaluate our proposed architecture on a synthetic data set, as well as on MNIST, Fashion MNIST, and CelebA. Our results show that RRAEs offer several advantages over Vanilla AEs with both large and small latent spaces, and outperform other regularizing AE architectures.

Rank Reduction Autoencoders

TL;DR

Rank Reduction Autoencoders (RRAEs) regularize latent spaces by integrating a truncated SVD into the forward pass, making the bottleneck a function of the latent-space rank rather than a fixed dimension. The adaptive variant aRRAEs automatically discovers the optimal bottleneck during training by pruning the least important singular value and redistributing its influence, without adding extra learning hyperparameters. Empirical results on synthetic data, MNIST, Fashion-MNIST, and CelebA show that RRAEs and aRRAEs yield stable training, scalable bottlenecks, and latent spaces conducive to interpolation and realistic sample generation, outperforming vanilla and other regularizing autoencoders. The work demonstrates that a rank-based bottleneck can provide more meaningful representations and simpler deployment, with practical impact on unsupervised representation learning and generative tasks, especially where data manifolds are complex or high-dimensional.

Abstract

The choice of an appropriate bottleneck dimension and the application of effective regularization are both essential for Autoencoders to learn meaningful representations from unlabeled data. In this paper, we introduce a new class of deterministic autoencoders, Rank Reduction Autoencoders (RRAEs), which regularize their latent spaces by employing a truncated singular value decomposition (SVD) during training. In RRAEs, the bottleneck is defined by the rank of the latent matrix, thereby alleviating the dependence of the encoder/decoder architecture on the bottleneck size. This approach enabled us to propose an adaptive algorithm (aRRAEs) that efficiently determines the optimal bottleneck size during training. We empirically demonstrate that both RRAEs and aRRAEs are stable, scalable, and reliable, as they do not introduce any additional training hyperparameters. We evaluate our proposed architecture on a synthetic data set, as well as on MNIST, Fashion MNIST, and CelebA. Our results show that RRAEs offer several advantages over Vanilla AEs with both large and small latent spaces, and outperform other regularizing AE architectures.
Paper Structure (18 sections, 1 theorem, 17 equations, 15 figures, 6 tables, 2 algorithms)

This paper contains 18 sections, 1 theorem, 17 equations, 15 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Autoencoders and Bottlenecks
  3. The Singular Value Decomposition (SVD)
  4. Rank Reduction Autoencoders (RRAEs) - Fixed Bottleneck
  5. RRAEs Without Batches
  6. RRAEs With Batches
  7. Adaptive Rank Reduction Auteoncoders (aRRAEs)
  8. Results and Discussion
  9. Synthetic Data Set
  10. Real World Data Sets
  11. Ablation Study
  12. Scalability of RRAEs
  13. Hyperparameter dependence
  14. Conclusion
  15. Gradients and the Stable SVD
  16. ...and 3 more sections

Key Result

Lemma 1

Let $A \in \mathbb{R}^{L \times L}$, if there exists at least $L$ linearly independent samples $Y_i\in\mathbb{R}^{L}$ for which $AY_i = 0$, then $A=[0]_{L\times L}$.

Figures (15)

  • Figure 1: Schematic illustrating RRAE's architecture when trained without batches.
  • Figure 2: Schematic illustrating why a latent vector $Y_i$ for a sample $X_i$ can be written differently when $X_i$ is processed with different batches. The colors are only to illustrate the dependence on a column.
  • Figure 3: Interpolation for different architectures on the 2D gaussian problem.
  • Figure 4: Random sampling for different architectures on the 2D gaussian problem.
  • Figure 5: Singular values of the latent space matrices for different architectures on the 2D gaussian problem. Log-scale to the right.
  • ...and 10 more figures

Theorems & Definitions (1)

  • Lemma 1