Deep Linear Discriminant Analysis Revisited

TL;DR

This work addresses the tension between generative modeling and discriminative performance in Deep LDA. It shows that unconstrained maximum-likelihood training can produce degenerate, poorly discriminative embeddings, while pure discriminative training with cross-entropy breaks the probabilistic interpretation of the LDA head. The authors propose the Discriminative Negative Log-Likelihood (DNLL), a simple penalty that augments the LDA likelihood with a density term to discourage high-overlap regions, thereby maintaining a coherent generative structure while preserving discrimination. On synthetic data and image benchmarks, DNLL yields clean latent spaces, competitive accuracy to softmax classifiers, and substantially improved probability calibration. The findings offer a practical pathway to principled deep discriminant models with reliable uncertainty estimates and suggest the DNLL loss as a scalable regularizer for probabilistic deep learning systems.

Abstract

We show that for unconstrained Deep Linear Discriminant Analysis (LDA) classifiers, maximum-likelihood training admits pathological solutions in which class means drift together, covariances collapse, and the learned representation becomes almost non-discriminative. Conversely, cross-entropy training yields excellent accuracy but decouples the head from the underlying generative model, leading to highly inconsistent parameter estimates. To reconcile generative structure with discriminative performance, we introduce the \emph{Discriminative Negative Log-Likelihood} (DNLL) loss, which augments the LDA log-likelihood with a simple penalty on the mixture density. DNLL can be interpreted as standard LDA NLL plus a term that explicitly discourages regions where several classes are simultaneously likely. Deep LDA trained with DNLL produces clean, well-separated latent spaces, matches the test accuracy of softmax classifiers on synthetic data and standard image benchmarks, and yields substantially better calibrated predictive probabilities, restoring a coherent probabilistic interpretation to deep discriminant models.

Figures (13)

• Figure 1: Deep LDA architecture. A neural encoder maps input data to a latent space where LDA is applied, and the network is optimized using the proposed Discriminative Negative Log-Likelihood (DNLL) loss.
• Figure 2: Classical three-class LDA fitted on synthetic data by maximum likelihood (solid ellipses, solid decision boundaries, means marked by '$\times$') and by minimizing cross-entropy with discriminant scores used as logits (dashed ellipses, dashed boundaries, means marked by '$+$'). The Gaussian components (means and covariances) differ noticeably, illustrating that cross-entropy training does not recover the maximum-likelihood parameters of the generative LDA model. Ellipses show the 90% contours of the learned Gaussian classes.
• Figure 3: Deep LDA embeddings $z_i=f_\psi(x_i)$ after likelihood training. Two classes merge into a single cluster; samples concentrate tightly around their class centroids $\{\mu_c\}$ while the shared covariance $\Sigma$ becomes nearly singular ($|\Sigma|\approx3\cdot10^{-10}$). Training accuracy: 67.2%; test accuracy: 67.7%.
• Figure 4: Deep LDA embeddings $f_\psi(x_i)$ after cross-entropy training. Gaussian components are located near the intersection of the boundaries rather than around the empirical class clouds yielding inconsistent parameter estimates. Training accuracy: 99.4%; test accuracy: 99.6%.
• Figure 5: Deep LDA embeddings $z_i = f_\psi(x_i)$ after training with the proposed Discriminative NLL. Unlike pure likelihood training (Figure \ref{['fig:deep_lda_emb']}), DNLL yields well-separated clusters with Gaussian components aligned to the empirical class clouds, while achieving near-perfect train and test accuracies: 99.4% and 99.5% respectively.

Theorems & Definitions (11)

• Theorem 1: Information potential for the LDA mixture (shared covariance)
• Remark 1: Diagonal vs. off-diagonal terms: repulsion and collapse
• proof : Proof of Theorem \ref{['thm:app-collision-lda']}
• Proposition 2: Control of overlap difference via KL
• proof
• Proposition 3: Explicit overlap bound for the LDA head
• proof
• Remark 2: The failure mode matches covariance collapse
• Theorem 4: Monotonicity of KL divergence under marginalization
• proof