Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint robust estimation

Xiang Li, Jun S. Liu, Qiang Sun, Lihu Xu

TL;DR

The paper tackles robust parameter estimation under heavy-tailed data by introducing joint estimators that solve two coupled Catoni-type equations to simultaneously recover trend parameters and the error variance for mean estimation, linear regression, and ridge regression. A key innovation is that these estimators cannot be derived from a single loss function, and their analysis leverages a Poincaré–Miranda framework to establish existence and non-asymptotic consistency with confidence intervals whose length scales as $Oig( oot 2 rac{ ext{log}(1/ ext{epsilon})}{n}ig)$. The contributions include tuning-free variance handling, flexibility in Catoni function choices, and explicit variance terms in the denominators that boost robustness to heavy tails, with rigorous non-asymptotic guarantees and practical validation via simulations and real-data experiments (e.g., Boston housing and NCI-60). The results demonstrate superior performance over traditional robust methods, especially under heavier tails or higher dimensions, and provide a general framework that can extend to other joint estimation problems in statistics under heavy-tailed noise.

Abstract

We introduce a joint robust estimation method for three parametric statistical models with heavy-tailed data: mean estimation, linear regression, and L2-penalized linear regression, where both the trend parameters and the error variance are unknown. Our approach is based on solving two coupled Catoni-type equations, one for estimating the trend parameters and the other for estimating the error variance. Notably, this joint estimation strategy cannot be obtained by minimizing a single loss function involving both the trend and variance parameters. The method offers four key advantages: (i) the length of the resulting (1 - epsilon) confidence interval scales as (log(1/epsilon))^{1/2}, matching the order achieved by classical estimators for sub-Gaussian data; (ii) it is tuning-free, eliminating the need for separate variance estimation; (iii) it allows flexible selection of Catoni-type functions tailored to the data; and (iv) it delivers strong performance for high-variance data, thanks to the explicit inclusion of the variance term in the denominators of both equations. We establish the consistency and asymptotic efficiency of the proposed joint robust estimators using new analytical techniques. The coupled equations are inherently complex, which makes the theoretical analysis of their solutions challenging. To address this, we employ the Poincare-Miranda theorem to show that the solutions lie within geometric regions, such as cylinders or cones, centered around the true parameter values. This methodology is of independent interest and extends to other statistical problems.

Joint robust estimation

TL;DR

The paper tackles robust parameter estimation under heavy-tailed data by introducing joint estimators that solve two coupled Catoni-type equations to simultaneously recover trend parameters and the error variance for mean estimation, linear regression, and ridge regression. A key innovation is that these estimators cannot be derived from a single loss function, and their analysis leverages a Poincaré–Miranda framework to establish existence and non-asymptotic consistency with confidence intervals whose length scales as . The contributions include tuning-free variance handling, flexibility in Catoni function choices, and explicit variance terms in the denominators that boost robustness to heavy tails, with rigorous non-asymptotic guarantees and practical validation via simulations and real-data experiments (e.g., Boston housing and NCI-60). The results demonstrate superior performance over traditional robust methods, especially under heavier tails or higher dimensions, and provide a general framework that can extend to other joint estimation problems in statistics under heavy-tailed noise.

Abstract

We introduce a joint robust estimation method for three parametric statistical models with heavy-tailed data: mean estimation, linear regression, and L2-penalized linear regression, where both the trend parameters and the error variance are unknown. Our approach is based on solving two coupled Catoni-type equations, one for estimating the trend parameters and the other for estimating the error variance. Notably, this joint estimation strategy cannot be obtained by minimizing a single loss function involving both the trend and variance parameters. The method offers four key advantages: (i) the length of the resulting (1 - epsilon) confidence interval scales as (log(1/epsilon))^{1/2}, matching the order achieved by classical estimators for sub-Gaussian data; (ii) it is tuning-free, eliminating the need for separate variance estimation; (iii) it allows flexible selection of Catoni-type functions tailored to the data; and (iv) it delivers strong performance for high-variance data, thanks to the explicit inclusion of the variance term in the denominators of both equations. We establish the consistency and asymptotic efficiency of the proposed joint robust estimators using new analytical techniques. The coupled equations are inherently complex, which makes the theoretical analysis of their solutions challenging. To address this, we employ the Poincare-Miranda theorem to show that the solutions lie within geometric regions, such as cylinders or cones, centered around the true parameter values. This methodology is of independent interest and extends to other statistical problems.

Paper Structure

This paper contains 28 sections, 12 theorems, 239 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Literature review
  3. Our contributions and methods
  4. Notation
  5. Joint robust estimations
  6. Joint robust mean and variance estimation
  7. Joint robust regression
  8. Joint robust ridge regression
  9. Simulations and real data analysis
  10. Numerical experiment 1: mean estimation
  11. Numerical experiment 2: linear regression
  12. Numerical experiment 3: $\ell_2$-penalized regression
  13. Real data analysis 1: Boston housing
  14. Real data analysis 2: NCI-60 Cancer Cell Lines
  15. A brief discussion on the selection of Catoni type functions
  16. ...and 13 more sections

Key Result

Theorem 2.1

Consider the joint estimation f1f2 with any confidence level $\epsilon\in (0, {1}/{4})$. There exists a constant $C=C(m_{2\beta},\sigma,\beta)$ such that, for all $n \ge C[\log(\epsilon^{-1})+1]$, f1f2 admits a solution $(\widehat{\theta},\widehat{v})$ satisfying, with probability at least $1-4\epsi and where $c_{0}=c_{0}(m_{2\beta},\sigma,\beta)$ does not depend on $n$ or $\epsilon$.

Figures (10)

  • Figure 1: The $\gamma$-quantile of the estimation error $|\mu-\widehat{\theta}|$ (estimation error, $y$-axis) versus $\gamma$ (quantile level, $x$-axis) for sample mean and three Catoni-type estimators based on 1000 simulations.
  • Figure 2: Empirical 99% quantile of the estimation error (99% quantile error, $y$-axis) for different methods versus a distribution parameter (standard deviation for normal, degrees of freedom for Student’s t, and shape for Pareto and Fréchet distributions, $x$-axis). The compared methods include the sample mean, Catoni’s estimator with true variance, Catoni’s estimator with sample variance, and the joint Catoni estimator.
  • Figure 3: The $\gamma$-quantile of the estimation error (estimation error $\|\widehat{\boldsymbol{\theta}}-\boldsymbol{\theta}^{*}\|_{2}$, $y$-axis) versus $\gamma$ (quantile level, $x$-axis) for OLS, adaptive Huber and joint Catoni estimators based on 1000 simulations with $n=500$ and $d=5$.
  • Figure 4: Empirical 99% quantile of the estimation error (99% quantile estimation error $\|\widehat{\boldsymbol{\theta}}-\boldsymbol{\theta}^{*}\|_{2}$, $y$-axis) for different methods versus a distribution parameter (standard deviation for normal, degrees of freedom for Student’s t, and shape for Pareto and Fréchet distributions, $x$-axis). The compared methods include OLS, adaptive Huber and joint Catoni estimators.
  • Figure 5: The $\gamma$ quantile of the estimation error (estimation error $\|\widehat{\boldsymbol{\theta}}-\boldsymbol{\theta}^{*}\|_{2}$, $y$-axis) versus $\gamma$ (quantile level, $x$-axis) for Ridge, $\ell_2$-penalized adaptive Huber and joint Catoni estimators based on 1000 simulations with $n=500$ and $d=200$.
  • ...and 5 more figures

Theorems & Definitions (25)

  • Theorem 2.1
  • Remark 2.2
  • Theorem 2.3
  • Theorem 2.4
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • Lemma B.1
  • Lemma B.2
  • ...and 15 more