Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Minimax Optimal Robust Sparse Regression with Heavy-Tailed Designs: A Gradient-Based Approach

Kaiyuan Zhou, Xiaoyu Zhang, Wenyang Zhang, Di Wang

TL;DR

This work tackles high-dimensional sparse regression under heavy-tailed noise and design by introducing RIGHT, a gradient-based robust framework that uses Median-of-Means gradients within Iterative Hard Thresholding. A key finding is a decoupling between estimation accuracy (driven by the noise tail index $\\delta$) and gradient stability (driven by the design tail index $\\lambda$), and the authors prove minimax lower bounds that align with the achieved rates. RIGHT attains near-minimax optimal estimation and sample complexity under weak moment conditions, without sample splitting or relying on a finite population risk; logistic regression, in particular, enjoys parametric rates due to bounded gradients, while multi-response regression extends the guarantees to matrix parameters. The theoretical results are complemented by extensive numerical studies and a real-data application, demonstrating the practical applicability of robust gradient-based sparse estimation in heavy-tailed settings.

Abstract

We investigate high-dimensional sparse regression when both the noise and the design matrix exhibit heavy-tailed behavior. Standard algorithms typically fail in this regime, as heavy-tailed covariates distort the empirical risk geometry. We propose a unified framework, Robust Iterative Gradient descent with Hard Thresholding (RIGHT), which employs a robust gradient estimator to bypass the need for higher-order moment conditions. Our analysis reveals a fundamental decoupling phenomenon: in linear regression, the estimation error rate is governed by the noise tail index, while the sample complexity required for stability is governed by the design tail index. This implies that while heavy-tailed noise limits precision, heavy-tailed designs primarily raise the sample size barrier for convergence. In contrast, for logistic regression, we show that the bounded gradient naturally robustifies the estimator against heavy-tailed designs, restoring standard parametric rates. We derive matching minimax lower bounds to prove that RIGHT achieves optimal estimation accuracy and sample complexity across these regimes, without requiring sample splitting or the existence of the population risk.

Minimax Optimal Robust Sparse Regression with Heavy-Tailed Designs: A Gradient-Based Approach

TL;DR

This work tackles high-dimensional sparse regression under heavy-tailed noise and design by introducing RIGHT, a gradient-based robust framework that uses Median-of-Means gradients within Iterative Hard Thresholding. A key finding is a decoupling between estimation accuracy (driven by the noise tail index ) and gradient stability (driven by the design tail index ), and the authors prove minimax lower bounds that align with the achieved rates. RIGHT attains near-minimax optimal estimation and sample complexity under weak moment conditions, without sample splitting or relying on a finite population risk; logistic regression, in particular, enjoys parametric rates due to bounded gradients, while multi-response regression extends the guarantees to matrix parameters. The theoretical results are complemented by extensive numerical studies and a real-data application, demonstrating the practical applicability of robust gradient-based sparse estimation in heavy-tailed settings.

Abstract

We investigate high-dimensional sparse regression when both the noise and the design matrix exhibit heavy-tailed behavior. Standard algorithms typically fail in this regime, as heavy-tailed covariates distort the empirical risk geometry. We propose a unified framework, Robust Iterative Gradient descent with Hard Thresholding (RIGHT), which employs a robust gradient estimator to bypass the need for higher-order moment conditions. Our analysis reveals a fundamental decoupling phenomenon: in linear regression, the estimation error rate is governed by the noise tail index, while the sample complexity required for stability is governed by the design tail index. This implies that while heavy-tailed noise limits precision, heavy-tailed designs primarily raise the sample size barrier for convergence. In contrast, for logistic regression, we show that the bounded gradient naturally robustifies the estimator against heavy-tailed designs, restoring standard parametric rates. We derive matching minimax lower bounds to prove that RIGHT achieves optimal estimation accuracy and sample complexity across these regimes, without requiring sample splitting or the existence of the population risk.
Paper Structure (48 sections, 25 theorems, 201 equations, 10 figures, 2 tables, 2 algorithms)

This paper contains 48 sections, 25 theorems, 201 equations, 10 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Sparse Regression under Heavy-Tailed Distributions
  3. Our Contributions
  4. Related Literature
  5. Notation and Outline
  6. Methodology
  7. Problem Setup
  8. Iterative Hard Thresholding For Sparse Estimation
  9. Median-of-Means for Robust Gradient Estimation
  10. Robust Iterative Gradient Descent with Hard Thresholding (RIGHT)
  11. Algorithm Convergence Analysis
  12. Application to Sparse Models
  13. Heavy-Tailed Linear Regression
  14. Heavy-Tailed Logistic Regression
  15. Heavy-Tailed Sparse Multi-Response Linear Regression
  16. ...and 33 more sections

Key Result

Proposition 3.2

If the population risk $\mathcal{R}(\boldsymbol{\theta})$ exists and satisfies $\beta$-restricted strong smoothness (RSS) and $\alpha$-restricted strong convexity (RSC), i.e., over the set of $(2s+s^*)$-sparse vectors $\boldsymbol{\theta}$, then the corresponding loss $\mathcal{L}$ satisfies the $(s,a,b)$-SRCG condition with $a=(2\beta)^{-1}$ and $b=\alpha/2$.

Figures (10)

  • Figure 1: Decoupled phase transitions in estimation error rate and sample complexity.
  • Figure 4: Linear regression performance comparison.
  • Figure 5: Logistic regression performance comparison.
  • Figure 6: Multi-response regression performance comparison.
  • Figure 7: Q-Q plot of responses and histogram of features kurtosis from data set riboflavin.
  • ...and 5 more figures

Theorems & Definitions (56)

  • Example 2.1: Linear Regression
  • Example 2.2: Gradients of Linear Regression
  • Remark 2.3: Element-wise MoM versus Multivariate Aggregation
  • Definition 3.1: Sparsity-Restricted Correlated Gradient, SRCG
  • Proposition 3.2: SRCG Implied by RSC/RSS
  • Definition 3.3: Sparsity-Restricted Stability (SRS) of Gradient Estimator
  • Remark 3.4
  • Theorem 3.5: Deterministic Convergence of RIGHT
  • Proposition 4.3: SRS of MoM Gradient Estimator for Linear Regression
  • Proposition 4.4: SRCG for Linear Regression
  • ...and 46 more