Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear regression with known noise distribution up to a scale: The reward of not using the OLSE

Fadoua Balabdaoui, Justine Leclerc

TL;DR

The paper studies linear regression when the error distribution is known up to a scale parameter and shows that the joint MLE of the regression vector and the scale is feasible, consistent, and asymptotically normal, with estimators that improve upon OLSE in efficiency. It derives the asymptotic relative efficiency, demonstrating that the MLE is at least as efficient as OLSE and often substantially more efficient for non-Gaussian noise, quantified for three scale-family distributions. Through simulations across multiple noise families and dimensions, and a real fish-market dataset, the authors illustrate the practical gains in estimation accuracy and variance reduction. The work provides a rigorous framework for ML-based estimation under scale-family noise, with clear guidelines on when leveraging the noise structure yields meaningful improvements in regression analysis.

Abstract

While the ordinary least squares estimator (OLSE) is still the most used estimator in linear regression models, other estimators can be more efficient when the error distribution is not Gaussian. In this paper, our goal is to evaluate this efficiency in the case of the Maximum Likelihood estimator (MLE) when the noise distribution belongs to a scale family. Under some regularity conditions, we show that (β_n,s_n), the MLE of the unknown regression vector β_0 and the scale s_0 exists and give the expression of the asymptotic efficiency of β_n over the OLSE. For given three scale families of densities, we quantify the true statistical gain of the MLE as a function of their deviation from the Gaussian family. To illustrate the theory, we present simulation results for different settings and also compare the MLE to the OLSE for the real market fish dataset.

Linear regression with known noise distribution up to a scale: The reward of not using the OLSE

TL;DR

The paper studies linear regression when the error distribution is known up to a scale parameter and shows that the joint MLE of the regression vector and the scale is feasible, consistent, and asymptotically normal, with estimators that improve upon OLSE in efficiency. It derives the asymptotic relative efficiency, demonstrating that the MLE is at least as efficient as OLSE and often substantially more efficient for non-Gaussian noise, quantified for three scale-family distributions. Through simulations across multiple noise families and dimensions, and a real fish-market dataset, the authors illustrate the practical gains in estimation accuracy and variance reduction. The work provides a rigorous framework for ML-based estimation under scale-family noise, with clear guidelines on when leveraging the noise structure yields meaningful improvements in regression analysis.

Abstract

While the ordinary least squares estimator (OLSE) is still the most used estimator in linear regression models, other estimators can be more efficient when the error distribution is not Gaussian. In this paper, our goal is to evaluate this efficiency in the case of the Maximum Likelihood estimator (MLE) when the noise distribution belongs to a scale family. Under some regularity conditions, we show that (β_n,s_n), the MLE of the unknown regression vector β_0 and the scale s_0 exists and give the expression of the asymptotic efficiency of β_n over the OLSE. For given three scale families of densities, we quantify the true statistical gain of the MLE as a function of their deviation from the Gaussian family. To illustrate the theory, we present simulation results for different settings and also compare the MLE to the OLSE for the real market fish dataset.

Paper Structure

This paper contains 35 sections, 5 theorems, 159 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation and background
  3. Linear model with known error distribution: Literature review
  4. Contributions and organization of the paper
  5. ML estimation with a scale-distribution for the noise
  6. Context and assumptions
  7. Existence
  8. Asymptotic normality of the MLE
  9. Consistency
  10. Weak convergence
  11. Asymptotic relative efficiency with respect to the OLSE
  12. Asymptotic efficiency
  13. Quantifying the gain for using the MLE under certain noise distributions
  14. Simulations and a data application
  15. Synthetic data
  16. ...and 20 more sections

Key Result

Theorem 1

Assume that (A4) holds. For any fixed $n \ge 1$, we assume further that Then, there exists al least one MLE $(\widehat{\beta}_n,\widehat{s}_n) \in \mathbb{R}^d \times (0, \infty)$.

Figures (10)

  • Figure 1: Semi-Supervised learning design for unmatched settings.
  • Figure 2: Asymptotic relative efficiency of the MLE versus the OLSE for the three families as a function of $\gamma$. $\gamma^*$ are the values for which it is maximized
  • Figure 3: Boxplots for the Euclidean norm of the estimator to the true $\beta$ (part 1).
  • Figure 4: Boxplots for the Euclidean norm of the estimator to the true $\beta$ (part 2).
  • Figure 5: Boxplots for the euclidean norm of the estimators to the true $\beta$.
  • ...and 5 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Proposition 1
  • Theorem 2
  • Theorem 3
  • Remark 1
  • Proposition 2
  • Definition 1
  • Definition 2