Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frequentist Oracle Properties of Bayesian Stacking Estimators

Valentin Zulj, Shaobo Jin, Måns Magnusson

Abstract

Compromise estimation entails using a weighted average of outputs from several candidate models, and is a viable alternative to model selection when the choice of model is not obvious. As such, it is a tool used by both frequentists and Bayesians, and in both cases, the literature is vast and includes studies of performance in simulations and applied examples. However, frequentist researchers often prove oracle properties, showing that a proposed average asymptotically performs at least as well as any other average comprising the same candidates. On the Bayesian side, such oracle properties are yet to be established. This paper considers Bayesian stacking estimators, and evaluates their performance using frequentist asymptotics. Oracle properties are derived for estimators stacking Bayesian linear and logistic regression models, and combined with Monte Carlo experiments that show Bayesian stacking may outperform the best candidate model included in the stack. Thus, the result is not only a frequentist motivation of a fundamentally Bayesian procedure, but also an extended range of methods available to frequentist practitioners.

Frequentist Oracle Properties of Bayesian Stacking Estimators

Abstract

Compromise estimation entails using a weighted average of outputs from several candidate models, and is a viable alternative to model selection when the choice of model is not obvious. As such, it is a tool used by both frequentists and Bayesians, and in both cases, the literature is vast and includes studies of performance in simulations and applied examples. However, frequentist researchers often prove oracle properties, showing that a proposed average asymptotically performs at least as well as any other average comprising the same candidates. On the Bayesian side, such oracle properties are yet to be established. This paper considers Bayesian stacking estimators, and evaluates their performance using frequentist asymptotics. Oracle properties are derived for estimators stacking Bayesian linear and logistic regression models, and combined with Monte Carlo experiments that show Bayesian stacking may outperform the best candidate model included in the stack. Thus, the result is not only a frequentist motivation of a fundamentally Bayesian procedure, but also an extended range of methods available to frequentist practitioners.

Paper Structure

This paper contains 17 sections, 2 theorems, 54 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Notation and Preliminaries
  3. Compromise estimation
  4. Jackknife model averaging
  5. Bayesian stacking
  6. Asymptotic optimality
  7. Linear models
  8. Logistic models
  9. Monte Carlo experiments
  10. Linear regression
  11. Results
  12. Logistic regression
  13. Results
  14. Conclusions
  15. Statement and proof of Lemma 1
  16. ...and 2 more sections

Key Result

Theorem 3.1

Suppose that $\hat{\boldsymbol{w}}_{ \text{BS}}$ is used to compute $\bar{\boldsymbol{\mu}} = \sum_k \hat{w}_k \mathbb E[\boldsymbol y|\boldsymbol{y}, \boldsymbol{X}, M_k]$, with $\mathbb E[\boldsymbol y|\boldsymbol{y}, \boldsymbol{X}, M_k]$ as given in Equation eq:postpred_kean. Then, given assumpt

Figures (2)

  • Figure 1: Results of simulations made using Zellner's $g$ (left) and the $\mathcal{T}$ prior (right). A solid line indicates that $n = 50$ observations have been used to fit the candidates, while a dashed line indicates a larger sample size of $n = 100$.
  • Figure 2: Results of logistic regression simulations made using normal priors (left) and $\mathcal{T}$ priors (right). A solid line indicates that $n = 50$ observations have been used to fit the candidates, while a dashed line indicates a larger sample size of $n = 100$.

Theorems & Definitions (8)

  • Theorem 3.1
  • proof
  • Remark 3.1.1
  • Remark 3.1.2
  • Remark 3.2.1
  • Remark 3.2.2
  • Lemma A.1
  • proof