$\ell_0$ Factor Analysis: A P-Stationary Point Theory

TL;DR

This work develops an ell_0-regularized, KL-divergence-based factor-analysis framework that decomposes the covariance into a low-rank part $\mathbf{L}$ and a sparse part $\mathbf{S}$ via $\boldsymbol{\Sigma}=\mathbf{L}+\mathbf{S}$ with $\mathbf{L}\succeq0$, $\mathbf{S}\succeq0$. The problem is posed as minimizing $\operatorname{tr}(\mathbf{L}) + C\|\mathbf{S}\|_0 + \mu \mathcal{D}_{\mathrm{KL}}(\boldsymbol{\Sigma}\|\check{\boldsymbol{\Sigma}})$ subject to $\boldsymbol{\Sigma}=\mathbf{L}+\mathbf{S}$ and positive semidefiniteness, and is shown to admit a minimizer with $f$ smooth and jointly strictly convex. The authors introduce a proximal-stationarity (P-stationarity) framework to characterize optimality for the nonconvex $\ell_0$ problem and design an ADMM algorithm with convergence analysis, including a subsequence convergence result; they prove that any convergent subsequence has a P-stationary limit, though global convergence remains open. Numerical experiments on synthetic and real data demonstrate robust recovery of the latent factor rank and sparse structure, outperforming $\ell_1$-based methods in identifying sparsity patterns and yielding competitive covariance estimates. This approach provides a practical, theory-backed method for covariance decomposition in factor analysis with explicit low-rank and sparse components.

Abstract

Factor Analysis is a widely used modeling technique for stationary time series which achieves dimensionality reduction by revealing a hidden low-rank plus sparse structure of the covariance matrix. Such an idea of parsimonious modeling has also been important in the field of systems and control. In this article, a nonconvex nonsmooth optimization problem involving the $\ell_0$ norm is constructed in order to achieve the low-rank and sparse additive decomposition of the sample covariance matrix. We establish the existence of an optimal solution and characterize these solutions via the concept of proximal stationary points. Furthermore, an ADMM algorithm is designed to solve the $\ell_0$ optimization problem, and a subsequence convergence result is proved under reasonable assumptions. Finally, numerical experiments demonstrate the effectiveness of our method in comparison with some alternatives in the literature.

Figures (6)

• Figure 1: The $\ell_{0}$ proximal operator on the real line.
• Figure 2: The performance of the $\ell_0$ ADMM in terms of three metrics when $\gamma$ takes the value of $10^{-2}$, $10^{-4}$, $10^{-6}$, and $10^{-8}$, respectively, with $N=1000$, the true $r=4$, $\mathrm{SNR}=6$ and the sparsity level of $\hat{\mathbf S}$ being $\|\hat{\mathbf S}\|_0/p^2=0.055$.
• Figure 3: Comparison of the sparse structure of the estimated $\mathbf S^*$ under the $\ell_0$ and $\ell_1$ regularizations across different conditions, when $N=1000$, $r=4$ and $\mathrm{SNR}=6$. (a) The true $\hat{\mathbf S}$ is diagonal (scaled identity). (b) The true $\hat{\mathbf S}$ is nondiagonal sparse.
• Figure 4: The subspace recovery accuracy of $\boldsymbol{\Gamma}$ for $N=200$, $500$, and $1000$ in two scenarios: when the true rank $r = 4$ and when $r = 10$, respectively, with the true $\hat{\mathbf S}$ being diagonal (scaled identity) and $\mathrm{SNR}=1$. The integer below each boxplot represents the total number of outliers (marked with red "+").
• Figure 5: The subspace recovery accuracy across various parameter configurations $(\mu,C)$ with $\rho=16$ fixed on a dataset of size $N=1000$, the true $r=4$ and $\mathrm{SNR}=1$.

Theorems & Definitions (28)

• Remark 1
• Proposition 1
• proof
• Remark 2
• Theorem 1
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3