Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uniformly most powerful tests in linear models

Razvan G. Romanescu

TL;DR

This work proves that in a standard multiple regression with nuisance parameters, the coefficient t-test is uniformly most powerful unbiased (UMPU) for testing a predictor's effect, using Neyman-structure arguments that do not rely on unbiased test statistics. It then shows that transforming predictors via Gram-Schmidt to an orthogonal basis can yield more powerful tests when predictors are correlated, by reparameterizing the model and exploiting a resulting t-distributed test statistic $V hicksim t_{n-p}$ under the null. The paper develops a comprehensive framework comparing Gram-Schmidt regression to standard multiple regression, discusses interpretability through SEM-like causal structures, and introduces a new power metric $ riangle$ that encapsulates multicollinearity and effect sizes. Through simulations and a real-data example on air pollution, it demonstrates practical power gains, especially when predictors are positively correlated, and provides guidance for study design and model choice. Overall, it articulates when and how Gram-Schmidt-based formulations can outperform naive regression in terms of statistical power and interpretability, with implications for experimental planning and causal inference in multivariate settings.

Abstract

In the multiple regression model we prove that the coefficient t-test for a variable of interest is uniformly most powerful unbiased, with the other parameters considered nuisance. The proof is based on the theory of tests with Neyman-structure and does not assume unbiasedness or linearity of the test statistic. We further show that the Gram-Schmidt decomposition of the design matrix leads to a family of regression model with potentially more powerful tests for the corresponding transformed regressors. Finally, we discuss interpretation and performance criteria for the Gram-Schmidt regression compared to standard multiple regression, and show how the power differential has major implications for study design.

Uniformly most powerful tests in linear models

TL;DR

This work proves that in a standard multiple regression with nuisance parameters, the coefficient t-test is uniformly most powerful unbiased (UMPU) for testing a predictor's effect, using Neyman-structure arguments that do not rely on unbiased test statistics. It then shows that transforming predictors via Gram-Schmidt to an orthogonal basis can yield more powerful tests when predictors are correlated, by reparameterizing the model and exploiting a resulting t-distributed test statistic under the null. The paper develops a comprehensive framework comparing Gram-Schmidt regression to standard multiple regression, discusses interpretability through SEM-like causal structures, and introduces a new power metric that encapsulates multicollinearity and effect sizes. Through simulations and a real-data example on air pollution, it demonstrates practical power gains, especially when predictors are positively correlated, and provides guidance for study design and model choice. Overall, it articulates when and how Gram-Schmidt-based formulations can outperform naive regression in terms of statistical power and interpretability, with implications for experimental planning and causal inference in multivariate settings.

Abstract

In the multiple regression model we prove that the coefficient t-test for a variable of interest is uniformly most powerful unbiased, with the other parameters considered nuisance. The proof is based on the theory of tests with Neyman-structure and does not assume unbiasedness or linearity of the test statistic. We further show that the Gram-Schmidt decomposition of the design matrix leads to a family of regression model with potentially more powerful tests for the corresponding transformed regressors. Finally, we discuss interpretation and performance criteria for the Gram-Schmidt regression compared to standard multiple regression, and show how the power differential has major implications for study design.

Paper Structure

This paper contains 18 sections, 3 theorems, 33 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Conditionally best tests in regression
  3. Related work
  4. Regression on an orthonormal set of predictors
  5. Transforming the predictor set via Gram--Schmidt
  6. Multiple regression on correlated predictors
  7. Explicit models for multicollinearity
  8. Ridge regression
  9. Interpretation of Gram--Schmidt and multiple regressions
  10. Special cases
  11. Power differences between parameterizations
  12. Applications
  13. Simulations of power
  14. Example: air pollution dataset
  15. Discussion
  16. ...and 3 more sections

Key Result

Theorem 1

Suppose we observe data vector $\mathbf{Y}$ from the multiple regression model $\bm{Y}=\beta_1 \mathbf{x}_{1}+\beta_2 \mathbf{x}_{2}+...+\beta_p \mathbf{x}_{p}+ \bm{\epsilon}$, where $\bm{\epsilon} \sim N(0,\sigma^2 I),$ and $\mathbf{x_1,x_2,...,x_p}$ are fixed covariates, for $p<n$. Assume further where $V=\frac{\sqrt{n-p}\, \mathbf{x}_p^\intercal \mathbf{Y}}{\sqrt{\mathbf{Y}^\intercal \mathbf{Y

Figures (1)

  • Figure 1: Power profiles for the first coefficient t-test under the naive, Gram--Schmidt, and ridge regression models, for different values of $\rho, p$ and $\sigma$. All tests are one-sided. The variance inflation factor and average $\Delta$ are shown, for each setting.

Theorems & Definitions (8)

  • Theorem 1
  • proof
  • proof
  • Proposition 3
  • proof
  • Definition 1
  • Theorem 4
  • proof