Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Connecting model-based and model-free approaches to linear least squares regression

Lutz Duembgen, Laurie Davies

TL;DR

This paper investigates how p-values from model-based linear regression analyses have precise model-free interpretations, framing the problem in terms of orthogonal invariance and Haar measure. It introduces equivalence regions that reinterpret confidence regions in a model-free context and derives concrete forms under Gaussian and Gaussian-sequence models, including connections to Beta and F distributions. The results cover both standard and composite null models and extend to sparse-signal inference, providing exact finite-sample-like guarantees and guidance for permutation-based and high-dimensional variable selection methods. Overall, the work bridges classical likelihood-based inference with data-centric, model-agnostic interpretations, offering rigorous tools for assessing regressor relevance in both low- and high-dimensional settings.

Abstract

In a regression setting with a response vector and given regressor vectors, a typical question is to what extent the response is related to these regressors, specifically, how well it can be approximated by a linear combination of the latter. Classical methods for this question are based on statistical models for the conditional distribution of the response, given the regressors. In the present paper it is shown that various p-values resulting from this model-based approach have also a purely data-analytic, model-free interpretation. This finding is derived in a rather general context. In addition, we introduce equivalence regions, a reinterpretation of confidence regions in the model-free context.

Connecting model-based and model-free approaches to linear least squares regression

TL;DR

This paper investigates how p-values from model-based linear regression analyses have precise model-free interpretations, framing the problem in terms of orthogonal invariance and Haar measure. It introduces equivalence regions that reinterpret confidence regions in a model-free context and derives concrete forms under Gaussian and Gaussian-sequence models, including connections to Beta and F distributions. The results cover both standard and composite null models and extend to sparse-signal inference, providing exact finite-sample-like guarantees and guidance for permutation-based and high-dimensional variable selection methods. Overall, the work bridges classical likelihood-based inference with data-centric, model-agnostic interpretations, offering rigorous tools for assessing regressor relevance in both low- and high-dimensional settings.

Abstract

In a regression setting with a response vector and given regressor vectors, a typical question is to what extent the response is related to these regressors, specifically, how well it can be approximated by a linear combination of the latter. Classical methods for this question are based on statistical models for the conditional distribution of the response, given the regressors. In the present paper it is shown that various p-values resulting from this model-based approach have also a purely data-analytic, model-free interpretation. This finding is derived in a rather general context. In addition, we introduce equivalence regions, a reinterpretation of confidence regions in the model-free context.

Paper Structure

This paper contains 28 sections, 2 theorems, 56 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. The F test and other methods revisited
  3. The model-based approach
  4. P-values for $H_o$.
  5. The model-free point of view
  6. Composite null models
  7. Equivalence regions
  8. General considerations
  9. Model-based approach.
  10. Model-free interpretation.
  11. Example \ref{['ex:F']} (continued)
  12. Inference on a sparse signal
  13. Special case: the Gaussian sequence model
  14. Final comments and further work
  15. Permutation tests.
  16. ...and 13 more sections

Key Result

Lemma 1

For arbitrary fixed, linearly independent vectors $\boldsymbol{y}, \boldsymbol{x}_1,\ldots,\boldsymbol{x}_p \in \mathbb{R}^n$ and stochastically independent standard Gaussian random vectors $\boldsymbol{x}_j^* \in \mathbb{R}^n$, $p_o < j \le p$, In particular,

Figures (2)

  • Figure 1: Distribution of $k_{\ell,\alpha}(\boldsymbol{y})$ in 2nd scenario.
  • Figure 2: Distribution of $\cos \angle(\boldsymbol{\mu},\boldsymbol{y})$ and $\cos \angle(\boldsymbol{\mu}, \hat{\boldsymbol{\mu}})$ in 1st scenario (left) and 2nd scenario (right).

Theorems & Definitions (5)

  • Lemma 1
  • Example 2: F test
  • Example 3: Multiple t tests
  • Example 4: Multiple F tests
  • Lemma 5