Divergence Maximizing Linear Projection for Supervised Dimension Reduction
Biao Chen, Joshua Kortje
TL;DR
This work tackles supervised dimension reduction for binary classification under a general Gaussian model by proposing two linear projection methods that maximize the KL divergence between projected class-conditional distributions. The large-μ and small-μ regimes yield complementary algorithms, with a regime-based rule guiding their use, and the analysis extends to multiclass cases under a common covariance, where multiclass LDA is shown to preserve pairwise KL divergences. Key contributions include unifying prior SDR approaches, establishing KL-optimality results for LDA under equal covariance, and providing practical procedures for regime selection and projection computation validated by numerical experiments. The methods rely only on first- and second-order statistics, are easy to implement, and offer interpretable subspaces with potential applicability to real-world Gaussian or near-Gaussian data; future work targets non-Gaussian settings and mixtures to broaden applicability.
Abstract
This paper proposes two linear projection methods for supervised dimension reduction using only the first and second-order statistics. The methods, each catering to a different parameter regime, are derived under the general Gaussian model by maximizing the Kullback-Leibler divergence between the two classes in the projected sample for a binary classification problem. They subsume existing linear projection approaches developed under simplifying assumptions of Gaussian distributions, such as these distributions might share an equal mean or covariance matrix. As a by-product, we establish that the multi-class linear discriminant analysis, a celebrated method for classification and supervised dimension reduction, is provably optimal for maximizing pairwise Kullback-Leibler divergence when the Gaussian populations share an identical covariance matrix. For the case when the Gaussian distributions share an equal mean, we establish conditions under which the optimal subspace remains invariant regardless of how the Kullback-Leibler divergence is defined, despite the asymmetry of the divergence measure itself. Such conditions encompass the classical case of signal plus noise, where both the signal and noise have zero mean and arbitrary covariance matrices. Experiments are conducted to validate the proposed solutions, demonstrate their superior performance over existing alternatives, and illustrate the procedure for selecting the appropriate linear projection solution.