Overspecified Mixture Discriminant Analysis: Exponential Convergence, Statistical Guarantees, and Remote Sensing Applications

TL;DR

The paper analyzes overspecified MDA where an unbalanced two-component Gaussian mixture is fitted per class to data generated from a single Gaussian. It proves that, in the population limit, EM converges exponentially fast to the Bayes risk, and in finite samples, misclassification error achieves the optimal rate of $O(\sqrt{d/n})$ with $O(\log(n/d))$ EM iterations. The analysis hinges on KL divergence contraction and a radial Polyak–Łojasiewicz inequality on a hypersurface where variances are determined by the current location parameters, with extensions to learned variances and unbalanced weights. Empirical validation on remote sensing datasets (Salinas-A and EuroSAT) demonstrates practical benefits of overspecified MDA, improving classification boundaries and multimodal class separation, thereby providing a principled justification for using overspecified mixtures in complex data contexts.

Abstract

This study explores the classification error of Mixture Discriminant Analysis (MDA) in scenarios where the number of mixture components exceeds those present in the actual data distribution, a condition known as overspecification. We use a two-component Gaussian mixture model within each class to fit data generated from a single Gaussian, analyzing both the algorithmic convergence of the Expectation-Maximization (EM) algorithm and the statistical classification error. We demonstrate that, with suitable initialization, the EM algorithm converges exponentially fast to the Bayes risk at the population level. Further, we extend our results to finite samples, showing that the classification error converges to Bayes risk with a rate $n^{-1/2}$ under mild conditions on the initial parameter estimates and sample size. This work provides a rigorous theoretical framework for understanding the performance of overspecified MDA, which is often used empirically in complex data settings, such as image and text classification. To validate our theory, we conduct experiments on remote sensing datasets.

Figures (8)

• Figure 1: Projections of MNIST DBLP:journals/spm/Deng12 images of '3' on a 2D plane with UMAP McInnes2018 (left) and examples of images from each cluster (right). Clustering was performed with HDBSCAN DBLP:conf/pakdd/CampelloMS13.
• Figure 2: Plot of $\mathop{\mathrm{D_{KL}}}\nolimits[\mathcal{N}(\mathbf{0},\mathbf{I}\parallel\mathcal{G}(\boldsymbol{\theta}_t,\sigma^2_t)$ versus iteration number $t$.
• Figure 3: Plot of the KL divergence $D_\text{KL}\left[\mathcal{N}(\mathbf{0},\mathbf{I})\parallel\mathcal{G}(\hat{\boldsymbol{\theta}}_T,\hat{\sigma}^2_T)\right]$ versus the sample size $n$. We consider the case $d=2$, and use the starting value $\boldsymbol{\theta}_0=(0.20, 0.05)$. For each $n$ we generate a sample of size $n$ from $\mathcal{N}(\mathbf{0},\mathbf{I})$ and use the EM algorithm to fit the balanced two-component mixture $\mathcal{G}(\boldsymbol{\theta},\sigma^2)$ to it. We repeat this process 10 times and report the average value of the KL divergence across those 10 runs. The slope of the fitted line is $-1.004$.
• Figure 4: Decision boundaries for LDA vs. MDA on the two most confused class pairs (Lettuce romaine 4wk vs 5wk; 5wk vs 6wk), projected onto a 2D PCA subspace of the spectral features. Red and blue points denote test pixels from each class. The black dashed line shows LDA’s linear boundary, while the shaded regions indicate MDA’s two-component decision regions.
• Figure 5: Ground-truth and predicted class label maps for the Salinas-A scene. Left: ground-truth map of the 6 classes; Center: LDA classification; Right: MDA classification. The MDA map corrects many of LDA’s errors, yielding a closer match to the true class distribution (especially at boundaries between confused classes).

Theorems & Definitions (33)

• Theorem 1
• Theorem 2
• Lemma 3
• Theorem 4
• proof
• Theorem 5
• proof : Proof of Theorem \ref{['thm:main']}
• Lemma 6
• proof
• proof