Adversarially Perturbed Precision Matrix Estimation

TL;DR

The paper introduces an adversarially perturbed framework for precision matrix estimation, linking perturbations to regularization and, in the $\ell_2$ case, to Wasserstein shrinkage estimators. It derives tractable convex reformulations for $\ell_2$ perturbations and a surrogate objective for $\ell_\infty$ perturbations, revealing a moment-adaptive regularization that promotes sparsity and adversarial robustness. Through rigorous asymptotic analysis, the authors identify regimes for the perturbation scale $\delta_n=\eta n^{-\gamma}$, showing zero bias when $\gamma>1/2$, a bias for $\gamma\le1/2$, and nonnormal behavior in the surrogate case at $\gamma=1/2$, along with sparsity-recovery guarantees. Numerical experiments on synthetic and real data demonstrate improved sparsity recovery and robustness, with practical improvements in downstream classification tasks such as linear discriminant analysis on a leukemia dataset.

Abstract

Precision matrix estimation is a fundamental topic in multivariate statistics and modern machine learning. This paper proposes an adversarially perturbed precision matrix estimation framework, motivated by recent developments in adversarial training. The proposed framework is versatile for the precision matrix problem since, by adapting to different perturbation geometries, the proposed framework can not only recover the existing distributionally robust method but also inspire a novel moment-adaptive approach to precision matrix estimation, proven capable of sparsity recovery and adversarial robustness. Notably, the proposed perturbed precision matrix framework is proven to be asymptotically equivalent to regularized precision matrix estimation, and the asymptotic normality can be established accordingly. The resulting asymptotic distribution highlights the asymptotic bias introduced by perturbation and identifies conditions under which the perturbed estimation can be unbiased in the asymptotic sense. Numerical experiments on both synthetic and real data demonstrate the desirable performance of the proposed adversarially perturbed approach in practice.

Figures (6)

• Figure 1: Accuracy and Sparsity Recovery on $AR(1)$ and $AR(2)$
• Figure 2: Accuracy and Sparsity Recovery on $AR(3)$ and $AR(4)$
• Figure 3: Accuracy and Sparsity Recovery on the Circle model and the Star model
• Figure 4: Adversarial Robustness on $AR(1)$ and $AR(2)$
• Figure 5: Adversarial Robustness on $AR(3)$ and $AR(4)$

Theorems & Definitions (25)

• Theorem 2.1: Convex Reformulations under $\ell_2$-perturbation
• Theorem 2.2: nguyen2022distributionally
• Theorem 2.3: Equivalence
• Proposition 3.1: Adversarial Robustness
• Corollary 3.2: Sparsity Recovery
• Remark 3.3
• Proposition 4.1: Regularization Effect
• Remark 4.2
• Example 4.3
• Theorem 5.1: Asymptotic Distribution