Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online and Offline Robust Multivariate Linear Regression

Antoine Godichon-Baggioni, Stephane S. Robin, Laure Sansonnet

TL;DR

This work develops robust estimation methods for multivariate Gaussian linear regression under both Euclidean and Mahalanobis loss criteria. It introduces online stochastic gradient algorithms (with averaging) and offline fixed-point algorithms for each loss, together with ridge-regularized variants, and proves convergence and asymptotic normality under weak conditions. A practical strategy is provided for unknown noise covariance $\\Sigma$ via Median Covariation Matrix (MCM) estimation and eigen-decomposition to yield a robust inverse covariance, enabling Mahalanobis-based robust regression in practice. Through extensive simulations, the authors demonstrate substantial robustness gains over classical least-squares across varying contamination levels and dimensions, and show favorable computational efficiency for online methods; all methods are implemented in the R package RobRegression. The theoretical results, empirical findings, and scalable online/offline algorithms offer a principled toolkit for robust multivariate regression and discriminant analysis in streaming contexts.

Abstract

We consider the robust estimation of the parameters of multivariate Gaussian linear regression models. To this aim we consider robust version of the usual (Mahalanobis) least-square criterion, with or without Ridge regularization. We introduce two methods each considered contrast: (i) online stochastic gradient descent algorithms and their averaged versions and (ii) offline fix-point algorithms. Under weak assumptions, we prove the asymptotic normality of the resulting estimates. Because the variance matrix of the noise is usually unknown, we propose to plug a robust estimate of it in the Mahalanobis-based stochastic gradient descent algorithms. We show, on synthetic data, the dramatic gain in terms of robustness of the proposed estimates as compared to the classical least-square ones. Well also show the computational efficiency of the online versions of the proposed algorithms. All the proposed algorithms are implemented in the R package RobRegression available on CRAN.

Online and Offline Robust Multivariate Linear Regression

TL;DR

This work develops robust estimation methods for multivariate Gaussian linear regression under both Euclidean and Mahalanobis loss criteria. It introduces online stochastic gradient algorithms (with averaging) and offline fixed-point algorithms for each loss, together with ridge-regularized variants, and proves convergence and asymptotic normality under weak conditions. A practical strategy is provided for unknown noise covariance via Median Covariation Matrix (MCM) estimation and eigen-decomposition to yield a robust inverse covariance, enabling Mahalanobis-based robust regression in practice. Through extensive simulations, the authors demonstrate substantial robustness gains over classical least-squares across varying contamination levels and dimensions, and show favorable computational efficiency for online methods; all methods are implemented in the R package RobRegression. The theoretical results, empirical findings, and scalable online/offline algorithms offer a principled toolkit for robust multivariate regression and discriminant analysis in streaming contexts.

Abstract

We consider the robust estimation of the parameters of multivariate Gaussian linear regression models. To this aim we consider robust version of the usual (Mahalanobis) least-square criterion, with or without Ridge regularization. We introduce two methods each considered contrast: (i) online stochastic gradient descent algorithms and their averaged versions and (ii) offline fix-point algorithms. Under weak assumptions, we prove the asymptotic normality of the resulting estimates. Because the variance matrix of the noise is usually unknown, we propose to plug a robust estimate of it in the Mahalanobis-based stochastic gradient descent algorithms. We show, on synthetic data, the dramatic gain in terms of robustness of the proposed estimates as compared to the classical least-square ones. Well also show the computational efficiency of the online versions of the proposed algorithms. All the proposed algorithms are implemented in the R package RobRegression available on CRAN.
Paper Structure (67 sections, 18 theorems, 161 equations, 21 figures, 8 tables)

This paper contains 67 sections, 18 theorems, 161 equations, 21 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Framework for robust regression
  3. Online estimation of $\beta^{*}$
  4. Online estimation of $\beta^{*}$ based on the minimization of $G$
  5. Online estimation of $\beta^{*}$ based on the minimization of $G_{\Sigma}$
  6. A comparison of these two approaches
  7. Offline estimation of $\beta^*$
  8. A fixed-point algorithm based on the minimization of $G$
  9. A fixed-point algorithm based on the minimization of $G_{\Sigma}$
  10. The robust regularized regression
  11. Online estimation of $\beta^*$ based on the minimization of $G_{\Sigma,\lambda}$
  12. Fixed-point algorithm based on the minimization of $G_{\Sigma,\lambda}$
  13. Complexity and comparison with existing methods
  14. Complexity
  15. Comparison with existing methods
  16. ...and 52 more sections

Key Result

Theorem 1

Let $F$ be the law of $X$. Suppose that $X$ admits a moment of order $1$ and that for any sequence $\left( \beta_{n} \right) \in \mathcal{M}_{q,p}(\mathbb{R})$ such that $\|\beta_{n} \|_{F} \underset{n\to +\infty}{\longrightarrow} + \infty$, $\mathbb{E} \left[ \left\| \beta_{n}X \right\|\right] \un where $P_{0}=\mathcal{N}(0, \Sigma)$ and let us denote by $\beta_{f,Q}^{*}$ the minimizer of the co

Figures (21)

  • Figure 1: Mean squared error for the estimation of the regression coefficients $\beta$ ($MSE(\beta)$) for OLS estimates without or with Ridge regularization for a sample size of $n = 1000$(with Student outliers). The $y$-axis is log-scaled. Legend: $\square$ = Naive, $\medcirc$ = Offline, $\triangle$ = Online, $+$ = RRRR. Number in each box = fraction $f$ of outliers (in %).
  • Figure 2: Mean squared error for the estimation of the regression coefficients $\beta$ ($MSE(\beta)$) for OLS estimates with or without Ridge regularization for a fraction of outliers of $f = 5\%$ (left) and $f = 28\%$ (right) (with Student outliers). The $y$-axis is log-scaled. Legend: $\square$ = Naive, $\medcirc$ = Offline, $\triangle$ = Online, $+$ = RRRR. Number in each box = sample size $n$.
  • Figure 3: AUC for the detection of outliers for the OLS estimates with or without Ridge regularization for a sample size of $n = 1000$(with Student outliers). Same legend as Figure \ref{['fig::mseBeta-n1000']}. The box $f = 0$ is empty as no outlier exists.
  • Figure 4: Comparison of the mean squared error for the estimation of the regression coefficients $\beta$ ($MSE(\beta)$) of OLS and WLS estimates for Offline and Online procedures without Ridge regularization for a sample size of $n = 1000$(with Student outliers). Legend: $\medcirc$ = OLS, $\triangle$ = WLS. Number in each box = fraction $f$ of outliers (in %).
  • Figure 5: Comparison of the empirical variances of the $p \times q = 100$ regression coefficients estimates $\widehat{\beta}_{jk}$ with the asymptotic variance, with a fraction of $16\%$ of outliers (with Student outliers). Top: robust OLS procedure, bottom: Robust WLS procedure. Left: $n = 100$, center: $n=1000$, right: $n=10000$. Legend: $\medcirc$ = offline, $\triangle$ = online.
  • ...and 16 more figures

Theorems & Definitions (29)

  • Theorem 1
  • Theorem 2
  • Remark
  • Remark
  • Theorem 3
  • Theorem 4
  • Corollary 1
  • Theorem 5
  • Corollary 2
  • Theorem 6
  • ...and 19 more