Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistically Significant Linear Regression Coefficients Solely Driven By Outliers In Finite-sample Inference

Felix Reichel

TL;DR

The paper investigates how outliers distort inference in linear regression, demonstrating that a single outlier can render an insignificant coefficient statistically significant in finite samples. Through simulations, it contrasts ordinary least squares with robust Hubers regression to illustrate mitigation of outlier influence and emphasizes regression diagnostics as essential for trustworthy inference. It further analyzes single-outlier tests based on residuals, discusses how the normality assumption affects finite-sample t-tests, and situates these issues within the Gauss–Markov framework. By surveying a broad suite of regression methods, the work links diagnostic practices to robustness and provides practical guidance for reliable inference in the presence of outliers.

Abstract

In this paper, we investigate the impact of outliers on the statistical significance of coefficients in linear regression. We demonstrate, through numerical simulation using R, that a single outlier can cause an otherwise insignificant coefficient to appear statistically significant. We compare this with robust Huber regression, which reduces the effects of outliers. Afterwards, we approximate the influence of a single outlier on estimated regression coefficients and discuss common diagnostic statistics to detect influential observations in regression (e.g., studentized residuals). Furthermore, we relate this issue to the optional normality assumption in simple linear regression [14], required for exact finite-sample inference but asymptotically justified for large n by the Central Limit Theorem (CLT). We also address the general dangers of relying solely on p-values without performing adequate regression diagnostics. Finally, we provide a brief overview of regression methods and discuss how they relate to the assumptions of the Gauss-Markov theorem.

Statistically Significant Linear Regression Coefficients Solely Driven By Outliers In Finite-sample Inference

TL;DR

The paper investigates how outliers distort inference in linear regression, demonstrating that a single outlier can render an insignificant coefficient statistically significant in finite samples. Through simulations, it contrasts ordinary least squares with robust Hubers regression to illustrate mitigation of outlier influence and emphasizes regression diagnostics as essential for trustworthy inference. It further analyzes single-outlier tests based on residuals, discusses how the normality assumption affects finite-sample t-tests, and situates these issues within the Gauss–Markov framework. By surveying a broad suite of regression methods, the work links diagnostic practices to robustness and provides practical guidance for reliable inference in the presence of outliers.

Abstract

In this paper, we investigate the impact of outliers on the statistical significance of coefficients in linear regression. We demonstrate, through numerical simulation using R, that a single outlier can cause an otherwise insignificant coefficient to appear statistically significant. We compare this with robust Huber regression, which reduces the effects of outliers. Afterwards, we approximate the influence of a single outlier on estimated regression coefficients and discuss common diagnostic statistics to detect influential observations in regression (e.g., studentized residuals). Furthermore, we relate this issue to the optional normality assumption in simple linear regression [14], required for exact finite-sample inference but asymptotically justified for large n by the Central Limit Theorem (CLT). We also address the general dangers of relying solely on p-values without performing adequate regression diagnostics. Finally, we provide a brief overview of regression methods and discuss how they relate to the assumptions of the Gauss-Markov theorem.
Paper Structure (32 sections, 34 equations, 4 figures, 1 table)

This paper contains 32 sections, 34 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Linear Regression and Statistical Inference
  3. OLS Estimation
  4. Distribution of the OLS Estimator
  5. Estimating Variance
  6. Standard Error and Hypothesis Testing
  7. The Effect of a Single Outlier on Regression Coefficients
  8. Single Outlier Tests for Linear Regression Models
  9. Internally Studentized Residuals
  10. Maximum Absolute Internally Studentized Residual
  11. Normalized Maximum Ordinary Residual
  12. Distributional Properties and Critical Values
  13. Upper Bound of Single Outlier Test Statistics
  14. On the Normality Assumption in Regression Analysis
  15. An Overview of Regression Methods
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Upper bounds of critical values for $R_n$ by sample size $n$ and significance level $\alpha$.
  • Figure 2: Diagnostic plots for the OLS model. The residuals appear homoscedastic (constant variance), symmetrically distributed, and independent. The Q-Q plot indicates approximate normality, validating the assumptions of OLS.
  • Figure 3: Diagnostic plots for OLS model including a high-leverage outlier. The residuals show clear distortion: heteroscedasticity, skewness, and heavy tails are evident. The Q-Q plot deviates significantly from the normal line, and the residuals vs. fitted plot shows a large residual corresponding to the outlier. This illustrates the breakdown of classical OLS assumptions.
  • Figure 4: Residuals vs. fitted values for the robust regression model applied to the data with an outlier. Compared to the standard OLS fit (A.2), the residuals are more evenly spread and the extreme influence of the outlier is visibly diminished. This confirms that the robust method effectively downweights the anomalous observation, preserving the integrity of the regression fit.