A Convex formulation for linear discriminant analysis

TL;DR

ConvexLDA rewrites linear discriminant analysis as a convex optimization problem that jointly minimizes within-class distance to class centroids and maximizes centroid dispersion via a log-determinant term, controlled by the hyperparameter $\lambda$. The objective $\mathcal{L} = \mathcal{L}_1 + \lambda \mathcal{L}_2$, with $\mathcal{L}_1 = \mathrm{Tr}[(\tilde{C}-X)(\tilde{C}-X)^T AA^T]$ and $\mathcal{L}_2 = -\log \det (A^T \hat{C}\hat{C}^T A + \gamma I)$, yields a convex problem over $A \in \mathbb{R}^{d\times p}$ that avoids inverting $S_w$ and eliminates pairwise distance computations. The authors prove convexity (via a convex $\mathcal{L}_1$ and a convex $\mathcal{L}_2$) and derive a gradient $\frac{\partial \mathcal{L}}{\partial A} = 2(\tilde{C}-X)(\tilde{C}-X)^T A - 2\lambda(\hat{C}\hat{C}^T A)(A^T \hat{C}\hat{C}^T A + \gamma I)^{-1}$ for efficient optimization. Empirical results across diverse data domains (including RNA-Seq, image, and single-cell data) show ConvexLDA outperforms Fisher LDA and many state-of-the-art LDA variants while providing global optimality and scalability. The work also analyzes the λ-driven trade-off between intra-class compactness and inter-class separation and discusses future directions such as robustness via $\ell_1$-norm, sparsity for biomarker discovery, and potential nonlinear extensions.

Abstract

We present a supervised dimensionality reduction technique called Convex Linear Discriminant Analysis (ConvexLDA). The proposed model optimizes a multi-objective cost function by balancing two complementary terms. The first term pulls the samples of a class towards its centroid by minimizing a sample's distance from its class-centroid in low dimensional space. The second term pushes the classes far apart by maximizing their hyperellipsoid scattering volume via the logarithm of the determinant (\textit{log det}) of the outer product matrix formed by the low-dimensional class-centroids. Using the negative of the \textit{log det}, we pose the final cost as a minimization problem, which balances the two terms using a hyper-parameter $λ$. We demonstrate that the cost function is convex. Unlike Fisher LDA, the proposed method doesn't require to compute the inverse of a matrix, hence avoiding any ill-conditioned problem where data dimension is very high, e.g. RNA-seq data. ConvexLDA doesn't require pair-wise distance calculation, making it faster and more easily scalable. Moreover, the convex nature of the cost function ensures global optimality, enhancing the reliability of the learned embedding. Our experimental evaluation demonstrates that ConvexLDA outperforms several popular linear discriminant analysis (LDA)-based methods on a range of high-dimensional biological data, image data sets, etc.

Figures (6)

• Figure 1: Plot of $\mathcal{L}_1$ and $\mathcal{L}_2$ across $\lambda$.
• Figure 2: Visualization of synthetic samples using Fisher's LDA (panel 'B') and Convex LDA (panel 'A') in two-dimensional embedding space. Panel 'C' presents the distance among each class pairs in original, LDA, Convex-LDA space. ConvexLDA is run setting $\lambda$ to 1.
• Figure 3: Visualization of toy data set in LDA and ConvexLDA space. In this experiment, we ran ConvexLDA with two different values of $\lambda$.
• Figure 4: Optimal value of $\lambda$ for ISOLET data for embedding dimension 10.
• Figure 5: Optimal value of $\lambda$ for MNIST data for embedding dimension 2.