Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Input Adaptive Bayesian Model Averaging

Yuli Slavutsky, Sebastian Salazar, David M. Blei

TL;DR

IA-BMA reframes model averaging as input-dependent Bayesian model selection, enabling per-input weighting of multiple predictors. It introduces an input-adaptive prior using an energy-based formulation and learns an amortized variational posterior to estimate $p(J=j|D,x)$, yielding a Bayes-optimal input-specific mixture $p(y|x,D)=\sum_j f_j(y|x)\,p(J(x)=j|D,x)$. Theoretical results establish strong likelihood-type guarantees for the posterior-weights predictor relative to any per-input single model, with penalties vanishing as the posterior sharpens. Empirically, IA-BMA improves accuracy and calibration across regression and classification tasks, including personalized cancer drug response, credit-card fraud detection, and standard UCI benchmarks, outperforming both non-adaptive baselines and existing adaptive methods. The approach is scalable, provides principled uncertainty quantification, and offers practical benefits for heterogeneous data settings.

Abstract

This paper studies prediction with multiple candidate models, where the goal is to combine their outputs. This task is especially challenging in heterogeneous settings, where different models may be better suited to different inputs. We propose input adaptive Bayesian Model Averaging (IA-BMA), a Bayesian method that assigns model weights conditional on the input. IA-BMA employs an input adaptive prior, and yields a posterior distribution that adapts to each prediction, which we estimate with amortized variational inference. We derive formal guarantees for its performance, relative to any single predictor selected per input. We evaluate IABMA across regression and classification tasks, studying data from personalized cancer treatment, credit-card fraud detection, and UCI datasets. IA-BMA consistently delivers more accurate and better-calibrated predictions than both non-adaptive baselines and existing adaptive methods.

Input Adaptive Bayesian Model Averaging

TL;DR

IA-BMA reframes model averaging as input-dependent Bayesian model selection, enabling per-input weighting of multiple predictors. It introduces an input-adaptive prior using an energy-based formulation and learns an amortized variational posterior to estimate , yielding a Bayes-optimal input-specific mixture . Theoretical results establish strong likelihood-type guarantees for the posterior-weights predictor relative to any per-input single model, with penalties vanishing as the posterior sharpens. Empirically, IA-BMA improves accuracy and calibration across regression and classification tasks, including personalized cancer drug response, credit-card fraud detection, and standard UCI benchmarks, outperforming both non-adaptive baselines and existing adaptive methods. The approach is scalable, provides principled uncertainty quantification, and offers practical benefits for heterogeneous data settings.

Abstract

This paper studies prediction with multiple candidate models, where the goal is to combine their outputs. This task is especially challenging in heterogeneous settings, where different models may be better suited to different inputs. We propose input adaptive Bayesian Model Averaging (IA-BMA), a Bayesian method that assigns model weights conditional on the input. IA-BMA employs an input adaptive prior, and yields a posterior distribution that adapts to each prediction, which we estimate with amortized variational inference. We derive formal guarantees for its performance, relative to any single predictor selected per input. We evaluate IABMA across regression and classification tasks, studying data from personalized cancer treatment, credit-card fraud detection, and UCI datasets. IA-BMA consistently delivers more accurate and better-calibrated predictions than both non-adaptive baselines and existing adaptive methods.
Paper Structure (45 sections, 2 theorems, 28 equations, 8 figures, 8 tables, 1 algorithm)

This paper contains 45 sections, 2 theorems, 28 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Input adaptive model averaging methods:
  4. Probabilistic formulation of adaptive model averaging
  5. Likelihood guarantees
  6. IA-BMA: Input Adaptive Bayesian Model Averaging
  7. Adaptive prior
  8. A two-model Bernoulli example
  9. Evaluation of the prior:
  10. Amortized Variational Posterior
  11. Optimization
  12. Weight assignment:
  13. Experiments
  14. Simulation
  15. Results:
  16. ...and 30 more sections

Key Result

Theorem 2.1

Denote $\mathcal{D}_{i} \coloneqq \{(x_t,y_t)\}_{t=1}^{i}$, and consider the posterior weights predictor $\hat{p}_\alpha^{(i)}$ assigning $\alpha_j(x; \mathcal{D}_{i})=p(J(x) =j \mid \mathcal{D}_{i},x)$ to the $j$-th predictor $f_j$. Assume that $\mathbb{E}[\vert \log f_j(Y\mid X)\vert]<\infty$ for where the expectations are taken w.r.t the population distribution $(x_i,y_i)\sim p(x,y)$.

Figures (8)

  • Figure 1: Illustration of the input adaptive prior. Each panel shows the posterior probabilities $p(J=j \mid \mathcal{D}, x)$ as functions of $x$. Top: the baseline log-odds is fixed and $\beta_1$ varies; larger $\beta_1$ values increase the influence of $x$, producing stronger adaptive corrections. Bottom: $\beta_1, \beta_2$ are fixed while the baseline log-odds $C_1 - C_2$ varies; stronger baselines yield higher prior preference for $f_1$, but input-specific corrections can still substantially reshape the prior at certain $x$. The marked point ($x=1$) highlights how the adaptive prior shifts the relative model probabilities compared to the baseline.
  • Figure 2: Simulation. Left: data (of one repetition). Results for accuracy (middle) and ECE (right) are reported for 10 repetitions. IA-BMA achieves highest accuracy and lowest ECE.
  • Figure 3: Experimental results for main case studies. Results are reported for 10 repetitions. IA-BMA achieves best results compared to all other averaging method on both case studies.
  • Figure 4: Cases where IA-BMA assigns the highest weight to Ridge.
  • Figure 5: Cases where IA-BMA assigns the highest weight to XGB.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Theorem 2.1
  • Theorem
  • proof