Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mallows-type model averaging: Non-asymptotic analysis and all-subset combination

Jingfu Peng

TL;DR

This work provides finite-sample guarantees for Mallows-type model averaging (MMA) when combining least-squares estimators across general candidate subspaces, establishing sharp and non-exact oracle inequalities under a finite fourth-moment assumption and milder conditions for asymptotic optimality. It reveals fundamental limits for all-subset MA and introduces a dimension-adaptive Adap procedure that attains minimax optimal risk relative to the all-subset benchmark, clarifying the ensemble role of model-selection procedures. The results connect MMA to classical thresholding and garrote methods, and demonstrate through simulations that the Adap estimator, along with thresholding-based MS procedures, can achieve optimal rates under both fixed and diverging predictor dimensions. Overall, the paper advances finite-sample theory for MA, clarifies the limitations of all-subset strategies, and delivers a practically implementable, theoretically justified method for dimension-adaptive weight learning in model averaging.

Abstract

Model averaging (MA) and ensembling play a crucial role in statistical and machine learning practice. When multiple candidate models are considered, MA techniques can be used to weight and combine them, often resulting in improved predictive accuracy and better estimation stability compared to model selection (MS) methods. In this paper, we address two challenges in combining least squares estimators from both theoretical and practical perspectives. We first establish several oracle inequalities for least squares MA via minimizing a Mallows' $C_p$ criterion under an arbitrary candidate model set. Compared to existing studies, these oracle inequalities yield faster excess risk and directly imply the asymptotic optimality of the resulting MA estimators under milder conditions. Moreover, we consider candidate model construction and investigate the problem of optimal all-subset combination for least squares estimators, which is an important yet rarely discussed topic in the existing literature. We show that there exists a fundamental limit to achieving the optimal all-subset MA risk. To attain this limit, we propose a novel Mallows-type MA procedure based on a dimension-adaptive $C_p$ criterion. The implicit ensembling effects of several MS procedures are also revealed and discussed. We conduct several numerical experiments to support our theoretical findings and demonstrate the effectiveness of the proposed Mallows-type MA estimator.

Mallows-type model averaging: Non-asymptotic analysis and all-subset combination

TL;DR

This work provides finite-sample guarantees for Mallows-type model averaging (MMA) when combining least-squares estimators across general candidate subspaces, establishing sharp and non-exact oracle inequalities under a finite fourth-moment assumption and milder conditions for asymptotic optimality. It reveals fundamental limits for all-subset MA and introduces a dimension-adaptive Adap procedure that attains minimax optimal risk relative to the all-subset benchmark, clarifying the ensemble role of model-selection procedures. The results connect MMA to classical thresholding and garrote methods, and demonstrate through simulations that the Adap estimator, along with thresholding-based MS procedures, can achieve optimal rates under both fixed and diverging predictor dimensions. Overall, the paper advances finite-sample theory for MA, clarifies the limitations of all-subset strategies, and delivers a practically implementable, theoretically justified method for dimension-adaptive weight learning in model averaging.

Abstract

Model averaging (MA) and ensembling play a crucial role in statistical and machine learning practice. When multiple candidate models are considered, MA techniques can be used to weight and combine them, often resulting in improved predictive accuracy and better estimation stability compared to model selection (MS) methods. In this paper, we address two challenges in combining least squares estimators from both theoretical and practical perspectives. We first establish several oracle inequalities for least squares MA via minimizing a Mallows' criterion under an arbitrary candidate model set. Compared to existing studies, these oracle inequalities yield faster excess risk and directly imply the asymptotic optimality of the resulting MA estimators under milder conditions. Moreover, we consider candidate model construction and investigate the problem of optimal all-subset combination for least squares estimators, which is an important yet rarely discussed topic in the existing literature. We show that there exists a fundamental limit to achieving the optimal all-subset MA risk. To attain this limit, we propose a novel Mallows-type MA procedure based on a dimension-adaptive criterion. The implicit ensembling effects of several MS procedures are also revealed and discussed. We conduct several numerical experiments to support our theoretical findings and demonstrate the effectiveness of the proposed Mallows-type MA estimator.
Paper Structure (38 sections, 8 theorems, 121 equations, 3 figures, 1 table)

This paper contains 38 sections, 8 theorems, 121 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Other related work
  4. Organization
  5. Problem setup
  6. Setup and notation
  7. MMA with general candidate models
  8. Construction of candidate models
  9. General candidate models
  10. Oracle inequalities
  11. Implications for AOP with general candidates
  12. Implications for all-nested MA
  13. All-subset candidate models
  14. Fundamental limit
  15. Attainability
  16. ...and 23 more sections

Key Result

Proposition 1

Suppose Assumption ass:4_moment holds. Then, for any candidate model set $\mathcal{M} \in \mathbf{M}(M_n)$ and any $\boldsymbol{\mu} \in \mathbb{R}^n$, there exists a constant $C>0$ such that where $\widehat{\boldsymbol{\mu}}_{\widehat{\mathbf{w}}_1|\mathcal{M}}$ is the MMA estimator defined in (eq:mahat).

Figures (3)

  • Figure 1: Risk ratio of the MMA estimator $\widehat{\boldsymbol{\mu}}_{\widehat{\mathbf{w}}_1|\mathcal{M}_G}$ under the polynomially and exponentially decaying coefficients. Results for $t$-distributed errors are shown in row (a), and those for Pareto-distributed errors are shown in row (b).
  • Figure 2: The risk ratios of the three competing methods under different signal decay scenarios. In each subfigure, the red dashed line in the bottom-right panel represents the curve of $2\log p$.
  • Figure 3: Risk comparison of six competing methods. Results for the ordered cases are presented in row (a), and those for the unordered cases are shown in row (b).

Theorems & Definitions (12)

  • Proposition 1
  • Theorem 1
  • Remark 1
  • Corollary 1
  • Remark 2
  • Corollary 2
  • Theorem 2
  • Remark 3
  • Theorem 3
  • Remark 4
  • ...and 2 more