The Adaptive $τ$-Lasso: Robustness and Oracle Properties

TL;DR

The paper tackles robust variable selection in high-dimensional sparse linear regression under gross contamination by introducing the adaptive $\tau$-Lasso, which combines a robust $\tau$-scale loss with an adaptive $\ell_1$ penalty. By replacing the standard penalty with adaptive weights $w_j=1/|\tilde{\beta}_j|^{\gamma}$ and using a robust $\tau$-scale, the estimator achieves both variable-selection consistency and asymptotic normality for active coefficients, without stringent design assumptions. The authors establish root-$n$ consistency, oracle properties, finite-sample breakdown points, and influence functions, and they validate robustness and accuracy through comprehensive simulations against competing methods across contaminated and clean data, showing reliable performance in high-dimensional sparse settings. The results suggest that adaptive $\tau$-Lasso is a practical and theoretically sound tool for robust sparse regression when data are prone to outliers and high-leverage points, with implications for applications in genomics, econometrics, and engineering where robust variable selection is essential.

Abstract

This paper introduces a new regularized version of the robust $τ$-regression estimator for analyzing high-dimensional datasets subject to gross contamination in the response variables and covariates. The resulting estimator, termed adaptive $τ$-Lasso, is robust to outliers and high-leverage points. It also incorporates an adaptive $\ell_1$-norm penalty term, which enables the selection of relevant variables and reduces the bias associated with large true regression coefficients. More specifically, this adaptive $\ell_1$-norm penalty term assigns a weight to each regression coefficient. For a fixed number of predictors $p$, we show that the adaptive $τ$-Lasso has the oracle property, ensuring both variable-selection consistency and asymptotic normality. Asymptotic normality applies only to the entries of the regression vector corresponding to the true support, assuming knowledge of the true regression vector support. We characterize its robustness by establishing the finite-sample breakdown point and the influence function. We carry out extensive simulations and observe that the class of $τ$-Lasso estimators exhibits robustness and reliable performance in both contaminated and uncontaminated data settings. We also validate our theoretical findings on robustness properties through simulations. In the face of outliers and high-leverage points, the adaptive $τ$-Lasso and $τ$-Lasso estimators achieve the best performance or match the best performances of competing regularized estimators, with minimal or no loss in terms of prediction and variable selection accuracy for almost all scenarios considered in this study. Therefore, the adaptive $τ$-Lasso and $τ$-Lasso estimators provide attractive tools for a variety of sparse linear regression problems, particularly in high-dimensional settings and when the data is contaminated by outliers and high-leverage points.

Figures (3)

• Figure 1: Plots of RMSE as a function of $y^{\star}$ outlier magnitude for each of the estimators under Scenario 1, averaged over 100 trials. Except for the non-robust estimator Lasso, the RMSE values of the remaining estimators for larger values of $y^{\star}$ do not exceed those of the remaining estimators for smaller values of $y^{\star}$. Moreover, both adaptive $\tau$-Lasso and $\tau$-Lasso show slightly better performance than other regularized robust estimators.
• Figure 2: Plots comparing estimation bias between adaptive $\tau$-Lasso and $\tau$-Lasso. As observed from the bias paths, $\tau$-Lasso shrinks non-zero coefficients more heavily toward zero than adaptive $\tau$-Lasso. For large values of $\lambda$, $\tau$-Lasso significantly overshrinks non-zero coefficients compared to adaptive $\tau$-Lasso and produces highly biased estimates, in particular for the large coefficients of $\boldsymbol{\beta}_0$. This behavior continues until $\lambda$ gets sufficiently large, at which point both estimators shrink all coefficients to zero. The right-hand panel (zoomed-in view) shows that when $\lambda$ is too small, overfitting occurs, which even causes slight overestimation for $\hat{\beta}_1$ in adaptive $\tau$-Lasso.
• Figure 3: Plots of influence function (IF) and standardized sensitivity curve (SC) of the adaptive $\tau$-Lasso estimator as a function of $\mathbf{z}_0=(y_0,\mathbf{x}_{[0]})$ for a one-dimensional toy example with regularization parameter $\lambda_n=0.1/n$. As predicted, the plotted IF and SC are almost identical and bounded across the entire plotted space, which indicates the correctness of our results about the influence function of the adaptive $\tau$-Lasso estimator derived in Theorem 7.