Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning

TL;DR

This study probes the training dynamics of a single-layer GAN model from the perspective of subspace learning, framing these GANs as a novel approach to this fundamental task, and systematically compares the efficacy of GAN-based methods against conventional approaches, both theoretically and empirically.

Abstract

Subspace learning is a critical endeavor in contemporary machine learning, particularly given the vast dimensions of modern datasets. In this study, we delve into the training dynamics of a single-layer GAN model from the perspective of subspace learning, framing these GANs as a novel approach to this fundamental task. Through a rigorous scaling limit analysis, we offer insights into the behavior of this model. Extending beyond prior research that primarily focused on sequential feature learning, we investigate the non-sequential scenario, emphasizing the pivotal role of inter-feature interactions in expediting training and enhancing performance, particularly with an uninformed initialization strategy. Our investigation encompasses both synthetic and real-world datasets, such as MNIST and Olivetti Faces, demonstrating the robustness and applicability of our findings to practical scenarios. By bridging our analysis to the realm of subspace learning, we systematically compare the efficacy of GAN-based methods against conventional approaches, both theoretically and empirically. Notably, our results unveil that while all methodologies successfully capture the underlying subspace, GANs exhibit a remarkable capability to acquire a more informative basis, owing to their intrinsic ability to generate new data samples. This elucidates the unique advantage of GAN-based approaches in subspace learning tasks.

Figures (7)

• Figure 1: ODE results for learning rate $\tilde{\tau} = 0.04, \tau = 0.2$ and four different noise levels, with $d=2$. The columns represent $\eta_{G} = \eta_{T} = 2, 1, 3, 4$ respectively. At $\eta = 5$ or higher, the generator is unable to learn anything. In all cases, the green and red represent the two diagonals of $\textbf{P}$, and the blue and yellow represent the two diagonals of $\textbf{Q}$. We see that the simulations do match the predicted ODE results.
• Figure 2: ODE results when initialized with off-diagonal entries. We focus on the case $\eta_{G} = \eta_{T} = 2$, as that noise level is seen above to be ideal for learning. Additionally, in all cases, $\tilde{\tau} = 0.04, \tau = 0.2$. The solid lines are with our approach, while the dashed lines are using the discriminator in solvable. From left to right, we use an initialization of $0.1, 0.01, 0.001, 0.0001$ for each component of the macroscopic states. It can be seen that our approach outperforms the single-feature discriminator in every case, with the gap becoming larger as the initialization approaches $0$.
• Figure 3: The graph shows the Grassmann distance over time on the Olivetti Faces dataset, for Oja's method (Blue) and the GAN model (Orange), as well as the single-feature GAN model (Green). We use the same hyperparameters as all previous experiments, measured with respect to a full PCA decomposition which acts as a surrogate for the true subspace.
• Figure 4: We provide results on the Olivetti Faces dataset, a well-known dataset. We show the top 16 learned features for all approaches at 3 stages of training: after the 1st epoch, the 200th epoch, and the end of training. We train all approaches for 500 epochs, equivalent to approximately 50 timesteps of simulated training. It can be clearly seen that while Oja's method learns quicker than the GAN model, eventually the GAN model outperforms it. Additionally, we see that the features learned by the GAN model are much more diverse and meaningful than those learned by Oja's method (whose learned features are more similar). For the single-feature GAN model, we can see that the learning is significantly slower, and never approaches anywhere close to the other two results.
• Figure 5: Comparison between the generator basis vectors learned by the multi-feature and single-feature discriminators on 36 features. The multi-feature model is trained for 1 epoch, while the single-feature model is trained for 5 epochs.

Theorems & Definitions (13)

• Definition 4.1
• Definition 4.2
• Theorem 4.3
• Lemma B.1
• Lemma B.2: Lemma 7 of solvable
• Lemma B.3: Lemma 2 of solvable
• proof
• Lemma B.4
• proof
• Lemma B.5