Outlier-robust additive matrix decomposition

TL;DR

This work introduces outlier-robust additive matrix decomposition (RTRMD) for least-squares trace regression when the parameter is the sum of a low-rank and a sparse matrix under adversarial label contamination. It develops three design properties—PP, IP, and MP—derived from product-process, Chevet, and multiplier inequalities to control the interplay between decomposition, contamination, and design-noise interaction. The proposed estimators use sorted-Huber losses with decomposable regularizers and achieve near-optimal rates $\mathsf{r}(n,d_{eff,r}) + \mathsf{r}(n,d_{eff,s}) + \sqrt{(1+\log(1/\delta))/n} + \varepsilon\log(1/\varepsilon)$, adaptively across $s,r,\varepsilon,\delta$, and with $\delta$-subgaussian guarantees even when noise depends on features. The paper also provides lower bounds and simulations confirming substantial improvements of sorted-Huber losses over classical Huber and validates practical performance in contaminated, high-dimensional settings.

Abstract

We study least-squares trace regression when the parameter is the sum of a $r$-low-rank matrix and a $s$-sparse matrix and a fraction $ε$ of the labels is corrupted. For subgaussian distributions and feature-dependent noise, we highlight three needed design properties, each one derived from a different process inequality: a "product process inequality", "Chevet's inequality" and a "multiplier process inequality". These properties handle, simultaneously, additive decomposition, label contamination and design-noise interaction. They imply the near-optimality of a tractable estimator with respect to the effective dimensions $d_{eff,r}$ and $d_{eff,s}$ of the low-rank and sparse components, $ε$ and the failure probability $δ$. The near-optimal rate is $\mathsf{r}(n,d_{eff,r}) + \mathsf{r}(n,d_{eff,s}) + \sqrt{(1+\log(1/δ))/n} + ε\log(1/ε)$, where $\mathsf{r}(n,d_{eff,r})+\mathsf{r}(n,d_{eff,s})$ is the optimal rate in average with no contamination. Our estimator is adaptive to $(s,r,ε,δ)$ and, for fixed absolute constant $c>0$, it attains the mentioned rate with probability $1-δ$ uniformly over all $δ\ge\exp(-cn)$. Without matrix decomposition, our analysis also entails optimal bounds for a robust estimator adapted to the noise variance. Our estimators are based on "sorted" versions of Huber's loss. We present simulations matching the theory. In particular, it reveals the superiority of "sorted" Huber's losses over the classical Huber's loss.

Figures (5)

• Figure 1: Huber vs "Sorted" Huber losses in sparse regression: $\sqrt{\texttt{MSE}}$ versus $\epsilon$.
• Figure 2: Robust sparse linear regression: different levels of sparsity (a) and comparisons between methods (b).
• Figure 3: Robust low-rank trace regression: different values of rank (a), and comparisons between methods (b,c).
• Figure 4: Trace regression with additive matrix decomposition with $\epsilon=0$
• Figure 5: Robust trace regression with additive decomposition: different values of rank/sparsity (a), and comparisons between methods (b,c).

Theorems & Definitions (83)

• Definition 1: Sorted Huber-type losses
• Remark 1: Huber loss versus "Sorted" Huber loss
• Theorem 2: Robust trace regression with additive matrix decomposition
• Proposition 1
• Theorem 3: Robust sparse/low-rank regression
• Theorem 4: $\sigma$-adaptive robust sparse/low-rank regression
• Remark 2
• Remark 3: Comparison with 2019chinot
• Remark 4: Comparison with 2019dalalyan:thompson
• Remark 5: Further references in robust sparse regression