Simplex Deep Linear Discriminant Analysis

TL;DR

Deep LDA trained with full freedom under maximum likelihood can collapse embeddings and class means, yielding high likelihood but poor discrimination. The authors fix class means to the vertices of a regular simplex and enforce a spherical shared covariance, enabling stable, likelihood-based training that yields well-separated latent clusters. Empirical results on synthetic data and image benchmarks (Fashion-MNIST, CIFAR-10, CIFAR-100) show competitive accuracy relative to softmax while producing interpretable, geometry-rich embeddings. This work provides a principled generative alternative to discriminative heads by injecting mild geometric structure that preserves probabilistic interpretability and practical performance.

Abstract

We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective. While classical LDA is a simple Gaussian model with linear decision boundaries, attaching an LDA head to a neural encoder raises the question of how to train the resulting deep classifier by maximum likelihood estimation (MLE). We first show that end-to-end MLE training of an unconstrained Deep LDA model ignores discrimination: when both the LDA parameters and the encoder parameters are learned jointly, the likelihood admits a degenerate solution in which some of the class clusters may heavily overlap or even collapse, and classification performance deteriorates. Batchwise moment re-estimation of the LDA parameters does not remove this failure mode. We then propose a constrained Deep LDA formulation that fixes the class means to the vertices of a regular simplex in the latent space and restricts the shared covariance to be spherical, leaving only the priors and a single variance parameter to be learned along with the encoder. Under these geometric constraints, MLE becomes stable and yields well-separated class clusters in the latent space. On images (Fashion-MNIST, CIFAR-10, CIFAR-100), the resulting Deep LDA models achieve accuracy competitive with softmax baselines while offering a simple, interpretable latent geometry that is clearly visible in two-dimensional projections.

Figures (8)

• Figure 1: Three-class LDA example with shared covariance. Ellipses that contain 90% of the probability for each of the three classes are shown. Faint colors show decision regions, and solid lines mark decision boundaries.
• Figure 2: Gradient-based ML fit of a three-class LDA model ($n=50$). Dashed ellipses show the 90% contours of the learned Gaussian classes, and filled "X" markers indicate the estimated class centers. The recovered parameters nearly coincide with the closed-form MLE.
• Figure 3: Deep LDA embeddings $z_i=f_\psi(x_i)$ after likelihood training. Two classes collapse; samples lie near their class centroids $\{\mu_c\}$ with a nearly singular shared covariance $\Sigma$ ($|\Sigma|\approx3\cdot10^{-10}$). Training accuracy: 67.1%; test accuracy: 66.6%.
• Figure 4: Deep LDA embeddings $f_\psi(x_i)$ after likelihood training with batchwise LDA re-estimation. Two class clusters collapse; embeddings concentrate near their centroids $\{\mu_c\}$ while the shared covariance becomes ill-conditioned ($|\Sigma|\approx10^{-8}$). Training accuracy: 67.8%; test accuracy: 66.6%.
• Figure 5: Regular simplex in $\mathbb{R}^3$ for $C=4$. The four fixed class means ${\mu_c}$ form a tetrahedron with edges of equal length; dashed lines indicate hidden edges in the 2D projection.