Asymptotic Characterisation of Robust Empirical Risk Minimisation Performance in the Presence of Outliers

TL;DR

This work examines in detail how performance depends on the loss function and on the degree of outlier corruption in the training set and identifies a region of parameters where the optimal performance of the Huber loss is identical to that of the $\ell_2$ loss, offering insights into the use cases of different loss functions.

Abstract

We study robust linear regression in high-dimension, when both the dimension $d$ and the number of data points $n$ diverge with a fixed ratio $α=n/d$, and study a data model that includes outliers. We provide exact asymptotics for the performances of the empirical risk minimisation (ERM) using $\ell_2$-regularised $\ell_2$, $\ell_1$, and Huber losses, which are the standard approach to such problems. We focus on two metrics for the performance: the generalisation error to similar datasets with outliers, and the estimation error of the original, unpolluted function. Our results are compared with the information theoretic Bayes-optimal estimation bound. For the generalization error, we find that optimally-regularised ERM is asymptotically consistent in the large sample complexity limit if one perform a simple calibration, and compute the rates of convergence. For the estimation error however, we show that due to a norm calibration mismatch, the consistency of the estimator requires an oracle estimate of the optimal norm, or the presence of a cross-validation set not corrupted by the outliers. We examine in detail how performance depends on the loss function and on the degree of outlier corruption in the training set and identify a region of parameters where the optimal performance of the Huber loss is identical to that of the $\ell_2$ loss, offering insights into the use cases of different loss functions.

Figures (13)

• Figure 1: Generalisation and estimation errors as a function of the sample complexity $\alpha$ for $\beta = 0$, in two regimes where ERM do not always achieve BO performance for large $\alpha$. (Left) Here we plot the excess generalisation error for $\epsilon = 0.6$, $\Delta_{\text{\tiny{IN}}} = 1$ and $\Delta_{\text{\tiny{OUT}}} = 0.5$, and for all losses we use the optimal value of $\lambda$. We plot two versions of the Huber loss, one with optimal scale parameter $a$ and the other with fixed $a=1$. We see that only the error of the $\ell_2$ and Huber loss with optimal scale $a$ (superimposed in the plot) vanishes at large sample complexity, reaching BO performance. For fixed-scale Huber loss, and $\ell_1$ loss, the error converges to a finite value. The parameters used for the simulations (dots) of BO are $d=1000$ averaged over $50$ samples, while for the ERM $d=200$ averaged over $1000$ samples. (Right) Here we plot the estimation error for $\epsilon = 0.3$, $\Delta_{\text{\tiny{IN}}} = 1.0$ and $\Delta_{\text{\tiny{OUT}}} = 5.0$, and for all losses we use optimal $(\lambda, a)$. We see that all losses converge to a finite value of the estimation error, while the BO error goes to zero for large sample complexity. The parameters used for BO are $d=4000$ averaged over $10$ samples, while for the ERM $d=200$ aveaged over $1000$ samples.
• Figure 2: Excess generalisation error (top panels) as a function of the percentage of outliers' $\epsilon$ (left) and of the outliers' variance $\Delta_{\text{\tiny{OUT}}}$ (right), along with the respective optimal hyper-parameters $(\lambda, a)$ (bottom panels). In both plots $\alpha = 10$, $\beta = 0$, $\Delta_{\text{\tiny{IN}}} = 1$, and for all numerical simulations (dots) have $d=200$, and are averaged over $1000$ samples. (Left) In this plot $\Delta_{\text{\tiny{OUT}}} = 5$. We observe that for small $\epsilon$ both the $\ell_2$ and the Huber loss converge to BO performance, while $\ell_1$ does not. (Right) In this plot $\epsilon = 0.3$. For $\Delta_{\text{\tiny{OUT}}} < \Delta_{\text{\tiny{IN}}}$ we notice that the Huber scale parameter diverges, so the Huber and $\ell_2$ loss become identical. Furthermore, the BO estimator has a non-monotonic behaviour as a function of $\Delta_{\text{\tiny{OUT}}}$, while ERM estimators are monotonic.
• Figure 3: Difference between the generalisation error of optimally-regularised $\ell_2$ and Huber loss losses as a function of $\epsilon$ and $\Delta_{\text{\tiny{OUT}}}$ for $\Delta_{\text{\tiny{IN}}} = 1$, $\beta = 0$. Each shaded region denotes where the two losses have a non-zero difference, i.e. where it is better to use the Huber loss, for different values of $\alpha$ as indicated by the labels. Focusing on the region of small $\epsilon$ and large $\Delta_{\text{\tiny{OUT}}}$, i.e. when dealing with just a few extremely noisy outliers, the plot suggests that having many samples (large $\alpha$) makes using the Huber loss not advantageous.
• Figure 4: Sketch of the dependence of the various quantities in GAMP algorithm.
• Figure 5: The points represent the mean and error of the mean of 16 instances of the simulations with data as described before. The simulations are performed with 30% of outliers ($\epsilon = 0.3$), with an additive noise variance of the inliers $\Delta_{\rm IN} = 1$ and correlation coefficient of the outliers $\beta = 0$. The $\ell_2$ regularisation parameter is chosen to be the one that minimises the estimation error in all three Figures. (Left) This plot is the counterpart to Figure \ref{['fig:figure-1']} (right) of the main text. Here we vary the sample complexity $\alpha$ and fix the noise variance of the outliers $\Delta_{\rm OUT} = 5$. Additionally for the Huber loss we fixed the scale parameter $a = 1$. (Center) & (Right) These two plots are the counterparts of Figure \ref{['fig:figure-2']} (right) of the main text. In these plots we are varying $\Delta_{\rm OUT}$ with a fixed sample complexity $\alpha = 10$.

Theorems & Definitions (5)

• Theorem 4.1
• Theorem 5.1
• Theorem 6.1
• Corollary A.1
• proof