Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Approximation Based Theory of Linear Regression

Laurie Davies

Abstract

The goal of this paper is to provide a theory linear regression based entirely on approximations. It will be argued that the standard linear regression model based theory whether frequentist or Bayesian has failed and that this failure is due to an 'assumed (revealed?) truth' (John Tukey) attitude to the models. This is reflected in the language of statistical inference which involves a concept of truth, for example efficiency, consistency and hypothesis testing. The motivation behind this paper was to remove the word `true' from the theory and practice of linear regression and to replace it by approximation. The approximations considered are the least squares approximations. An approximation is called valid if it contains no irrelevant covariates. This is operationalized using the concept of a Gaussian P-value which is the probability that pure Gaussian noise is better in term of least squares than the covariate. The precise definition given in the paper is intuitive and requires only four simple equations. Given this a valid approximation is one where all the Gaussian P-values are less than a threshold $p0$ specified by the statistician, in this paper with the default value 0.01. This approximations approach is not only much simpler it is overwhelmingly better than the standard model based approach. This will be demonstrated using six real data sets, four from high dimensional regression and two from vector autoregression. Both the simplicity and the superiority of Gaussian P-values derive from their universal exactness and validity. This is in complete contrast to standard F P-values which are valid only for carefully designed simulations. The paper contains excerpts from an unpublished paper by John Tukey entitled `Issues relevant to an honest account of data-based inference partially in the light of Laurie Davies's paper'.

An Approximation Based Theory of Linear Regression

Abstract

The goal of this paper is to provide a theory linear regression based entirely on approximations. It will be argued that the standard linear regression model based theory whether frequentist or Bayesian has failed and that this failure is due to an 'assumed (revealed?) truth' (John Tukey) attitude to the models. This is reflected in the language of statistical inference which involves a concept of truth, for example efficiency, consistency and hypothesis testing. The motivation behind this paper was to remove the word `true' from the theory and practice of linear regression and to replace it by approximation. The approximations considered are the least squares approximations. An approximation is called valid if it contains no irrelevant covariates. This is operationalized using the concept of a Gaussian P-value which is the probability that pure Gaussian noise is better in term of least squares than the covariate. The precise definition given in the paper is intuitive and requires only four simple equations. Given this a valid approximation is one where all the Gaussian P-values are less than a threshold specified by the statistician, in this paper with the default value 0.01. This approximations approach is not only much simpler it is overwhelmingly better than the standard model based approach. This will be demonstrated using six real data sets, four from high dimensional regression and two from vector autoregression. Both the simplicity and the superiority of Gaussian P-values derive from their universal exactness and validity. This is in complete contrast to standard F P-values which are valid only for carefully designed simulations. The paper contains excerpts from an unpublished paper by John Tukey entitled `Issues relevant to an honest account of data-based inference partially in the light of Laurie Davies's paper'.
Paper Structure (42 sections, 14 equations, 6 figures, 17 tables)

This paper contains 42 sections, 14 equations, 6 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Gaussian P-values and approximations
  3. Gaussian P-values
  4. Approximations
  5. Functions based on Gaussian P-values
  6. The function f1st
  7. The function f2st
  8. The function f3st
  9. Two examples using f3st
  10. Further examples: the leukemia, the osteoarthritis, the Boston housing and the sunspot data sets
  11. The leukemia data
  12. The osteoarthritis data
  13. The Boston housing data 1
  14. The Boston housing data 2
  15. The Boston housing data 3
  16. ...and 27 more sections

Figures (6)

  • Figure 1: Top: data and the approximation (red). Bottom: the residuals.
  • Figure 2: Top: USA GDP and the approximation (red). Bottom: the residuals.
  • Figure 3: Top:data and the non-parametric regression function. Bottom: empirical distribution function of autoregessive error term divided by 16 with $F=t_3$.
  • Figure 4: Plot of $-log(gpvalue)$ for each of the 190 covariates in the approximations to the riboflavin data.
  • Figure 5: This is an undirected dependency graph for the 32 covariates of Table \ref{['tab:frequencies']}.
  • ...and 1 more figures