Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exponential Lasso: robust sparse penalization under heavy-tailed noise and outliers with exponential-type loss

The Tien Mai

TL;DR

The paper tackles robust sparse regression in high dimensions by replacing the Lasso's squared loss with an exponential-type loss, yielding the Exponential Lasso. It develops a Majorization-Minimization algorithm that solves a sequence of weighted Lasso problems, with theoretical non-asymptotic guarantees under weak noise assumptions. Empirical results show strong robustness to heavy-tailed noise and outliers, while remaining competitive under Gaussian noise, supported by real-data applications in cancer genomics. The work provides practical guidance, a public R package heavylasso, and highlights future directions for parameter tuning and extension to other models.

Abstract

In high-dimensional statistics, the Lasso is a cornerstone method for simultaneous variable selection and parameter estimation. However, its reliance on the squared loss function renders it highly sensitive to outliers and heavy-tailed noise, potentially leading to unreliable model selection and biased estimates. To address this limitation, we introduce the Exponential Lasso, a novel robust method that integrates an exponential-type loss function within the Lasso framework. This loss function is designed to achieve a smooth trade-off between statistical efficiency under Gaussian noise and robustness against data contamination. Unlike other methods that cap the influence of large residuals, the exponential loss smoothly redescends, effectively downweighting the impact of extreme outliers while preserving near-quadratic behavior for small errors. We establish theoretical guarantees showing that the Exponential Lasso achieves strong statistical convergence rates, matching the classical Lasso under ideal conditions while maintaining its robustness in the presence of heavy-tailed contamination. Computationally, the estimator is optimized efficiently via a Majorization-Minimization (MM) algorithm that iteratively solves a series of weighted Lasso subproblems. Numerical experiments demonstrate that the proposed method is highly competitive, outperforming the classical Lasso in contaminated settings and maintaining strong performance even under Gaussian noise. Our method is implemented in the \texttt{R} package \texttt{heavylasso} available on Github: https://github.com/tienmt/heavylasso

Exponential Lasso: robust sparse penalization under heavy-tailed noise and outliers with exponential-type loss

TL;DR

The paper tackles robust sparse regression in high dimensions by replacing the Lasso's squared loss with an exponential-type loss, yielding the Exponential Lasso. It develops a Majorization-Minimization algorithm that solves a sequence of weighted Lasso problems, with theoretical non-asymptotic guarantees under weak noise assumptions. Empirical results show strong robustness to heavy-tailed noise and outliers, while remaining competitive under Gaussian noise, supported by real-data applications in cancer genomics. The work provides practical guidance, a public R package heavylasso, and highlights future directions for parameter tuning and extension to other models.

Abstract

In high-dimensional statistics, the Lasso is a cornerstone method for simultaneous variable selection and parameter estimation. However, its reliance on the squared loss function renders it highly sensitive to outliers and heavy-tailed noise, potentially leading to unreliable model selection and biased estimates. To address this limitation, we introduce the Exponential Lasso, a novel robust method that integrates an exponential-type loss function within the Lasso framework. This loss function is designed to achieve a smooth trade-off between statistical efficiency under Gaussian noise and robustness against data contamination. Unlike other methods that cap the influence of large residuals, the exponential loss smoothly redescends, effectively downweighting the impact of extreme outliers while preserving near-quadratic behavior for small errors. We establish theoretical guarantees showing that the Exponential Lasso achieves strong statistical convergence rates, matching the classical Lasso under ideal conditions while maintaining its robustness in the presence of heavy-tailed contamination. Computationally, the estimator is optimized efficiently via a Majorization-Minimization (MM) algorithm that iteratively solves a series of weighted Lasso subproblems. Numerical experiments demonstrate that the proposed method is highly competitive, outperforming the classical Lasso in contaminated settings and maintaining strong performance even under Gaussian noise. Our method is implemented in the \texttt{R} package \texttt{heavylasso} available on Github: https://github.com/tienmt/heavylasso

Paper Structure

This paper contains 27 sections, 4 theorems, 47 equations, 1 figure, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model and method
  3. Robust Lasso with Exponential-type loss
  4. Robustness of the loss
  5. Some intuitions:
  6. The Influence Function:
  7. Other insights:
  8. Statistical guarantee
  9. Discussion on noise assumptions.
  10. Majorization–Minimization Algorithm
  11. Algorithm development
  12. Coordinate descent updates using soft-thresholding
  13. Discussion.
  14. Convergence analysis of the MM algorithm
  15. Tuning the regularization parameter $\lambda$
  16. ...and 12 more sections

Key Result

Theorem 1

Under Assumptions asm:design--asm:noise, fix $\delta\in(0,1)$. Choose the tuning parameter Assume $r>0$ in Assumption asm:design is small enough so that for all $\Delta$ in the cone $\{\|\Delta\|_2\le r,\ \|\Delta_{S^c}\|_1\le 3\|\Delta_S\|_1\}$ it holds that $|x_i^\top\Delta|\le c/2$ for all $i$ (this is satisfied when $r$ is chosen such that $K\sqrt{s}\, r \le c/2$). Then with probabili where t

Figures (1)

  • Figure 1: Comparison of our loss function ($\tau = 0.5$) with other common losses: squared loss, absolute ($\ell_1$) loss, Tukey’s biweight loss, and Huber loss. The plot illustrates that, unlike Huber loss, our loss is much less sensitive to large residuals while closely resembling the squared loss for small residual values. Left: full-scale plot. Right: zoomed-in view near zero residuals.

Theorems & Definitions (7)

  • Theorem 1
  • Theorem 2: Monotone decrease and boundedness
  • Theorem 3: Cluster points are stationary
  • Corollary 1: Convergence of objective and subsequential stationarity
  • proof : Proof of Theorem \ref{['thm:correntropy_lasso']}
  • proof : Proof of Theorem \ref{['thm:descent']}
  • proof : Proof of Theorem \ref{['thm:stationary']}