From Model Selection to Model Averaging: A Comparison for Nested Linear Models

TL;DR

This work extends the MA vs MS comparison from Peng & Yang (2021) to a general nested linear regression framework that permits non-orthogonal designs, heteroscedastic and autocorrelated errors, and sparse coefficients. It introduces grouped variable importance via the GVI metric $\theta_{n,m}$, analyzes how the decay of $\theta_{n,m}$ and the number of candidate models $M_n$ govern when MA can offer meaningful gains over MS, and compares MA under three weight sets $\mathcal{W}_n$, $\mathcal{Q}_n$, and $\mathcal{W}_n(N)$. The paper establishes that MA never drastically surpasses MS in risk, but exhibits a phase-transition behavior: with large $M_n$ and slowly decaying $\theta_{n,m}$, MA can substantially reduce risk; with small $M_n$ or rapidly decaying $\theta_{n,m}$, MS and MA are asymptotically equivalent. It also characterizes how relaxing or discretizing the weight sets affects MA performance, provides two illustrative examples and extensive simulation studies, and delivers detailed proofs in the appendices. Overall, the results guide practitioners on when MA is advantageous and how weight-set choices influence practical performance.

Abstract

Model selection (MS) and model averaging (MA) are two popular approaches when having many candidate models. Theoretically, the estimation risk of an oracle MA is not larger than that of an oracle MS because the former one is more flexible, but a foundational issue is: does MA offer a {\it substantial} improvement over MS? Recently, a seminal work: Peng and Yang (2021), has answered this question under nested models with linear orthonormal series expansion. In the current paper, we further reply this question under linear nested regression models. Especially, a more general nested framework, heteroscedastic and autocorrelated random errors, and sparse coefficients are allowed in the current paper, which is more common in practice. In addition, we further compare MAs with different weight sets. Simulation studies support the theoretical findings in a variety of settings.

Figures (8)

• Figure 1: Two typical situations of the squared prediction risk $R_n(m)$ and the relationship between $M_n$ and $m_n^{**}$ under Assumption 6. In left panel: $M_n<m_n^{**}$. In right penal: $M_n\geq m_n^{**}$.
• Figure 2: Numerical illustration for Example 5.1 with $\alpha=0.8$. Left: plots of $\lim_{n\to \infty} {R_n(\bm{w}_n^*)\over R_n(\bm{w}_{n,N}^*)}$ against $N\in \{1,\ldots,10\}$ for $\kappa=0.5, 1, 2$, respectively. Right: plots of $\lim_{n\to \infty} {R_n(\bm{w}_{n,dN}^*)\over R_n(\bm{w}_{n,d}^*)}$ againsts $\kappa\in (0, 8)$ for $N=1, 2, 4$, respectively.
• Figure 3: Simulation results for Example 1 with the case of slowly decaying $\theta_m^*$. Normalized risk functions for AIC, BIC, LOO-CV, and MMA when $\theta_m^*=m^{-2\alpha_1}/\sigma^2$ with $\alpha_1=1$ in row (a), $\alpha_1=1.5$ in row (b), and $\alpha_1=2$ in row (c).
• Figure 4: Simulation results for Example 1 with the case of fast decaying $\theta_m^*$. Normalized risk functions for AIC, BIC, LOO-CV, and MMA when $\theta_m^*=\exp(-2\alpha_2 m)/\sigma^2$ with $\alpha_2=1$ in row (a), $\alpha_2=1.5$ in row (b), and $\alpha_2=2$ in row (c).
• Figure 5: Simulation results for Example 2 with the case of slowly decaying $\theta_m^*$. Normalized risk functions for AIC, BIC, LOO-CV, JMA2, and JMA when $\theta_m^*=c^2 m^{-2\alpha_1}$ with $\alpha_1=1$ in row (a), $\alpha_1=1.5$ in row (b), and $\alpha_1=2$ in row (c).

Theorems & Definitions (5)

• Remark 1
• Remark 2: Another explanation for $\theta_{n,m}$
• Lemma 1
• proof
• proof