AI-Powered Bayesian Inference

TL;DR

This paper proposes AI-powered Bayesian inference by treating generative AI outputs as priors on the data-generating distribution via a Dirichlet process, enabling coherent uncertainty quantification that leverages AI predictions without fully trusting them for decision making. It develops likelihood-driven AI priors (Power, Expected-Posterior, Catalytic) and a loss-based non-parametric framework (Bayes without likelihood) that places priors on the data-generating distribution $F_0$ and uses posterior bootstrap for computation. Theoretical results center on the concentration parameter $\alpha$, including coverage-based and asymptotic calibration to align AI-prior influence with frequentist properties. Empirical illustrations in dermatology and astronomy demonstrate improved predictive calibration and narrower uncertainty regions when AI priors are appropriately tuned, while highlighting the danger of over-reliance on AI priors as $\alpha$ grows large. Overall, the work offers a flexible, scalable, and principled route to integrate AI-driven predictions into Bayesian inference with quantified uncertainty across diverse domains.

Abstract

The advent of Generative Artificial Intelligence (GAI) has heralded an inflection point that changed how society thinks about knowledge acquisition. While GAI cannot be fully trusted for decision-making, it may still provide valuable information that can be integrated into a decision pipeline. Rather than seeing the lack of certitude and inherent randomness of GAI as a problem, we view it as an opportunity. Indeed, variable answers to given prompts can be leveraged to construct a prior distribution which reflects assuredness of AI predictions. This prior distribution may be combined with tailored datasets for a fully Bayesian analysis with an AI-driven prior. In this paper, we explore such a possibility within a non-parametric Bayesian framework. The basic idea consists of assigning a Dirichlet process prior distribution on the data-generating distribution with AI generative model as its baseline. Hyper-parameters of the prior can be tuned out-of-sample to assess the informativeness of the AI prior. Posterior simulation is achieved by computing a suitably randomized functional on an augmented data that consists of observed (labeled) data as well as fake data whose labels have been imputed using AI. This strategy can be parallelized and rapidly produces iid samples from the posterior by optimization as opposed to sampling from conditionals. Our method enables (predictive) inference and uncertainty quantification leveraging AI predictions in a coherent probabilistic manner.

Figures (4)

• Figure 1: Classification accuracy of ESD on held-out test data, using a neural network trained on $n=58$ observations. Each line indicates the mean performance after 10 repetitions. Horizontal lines indicate the test performance of a fitted neural network fit on training data only, of ChatGPTs imputations, and of a neural networked fit only on imputed data.
• Figure 2: 90% Credible intervals for estimating the proportion of spiral galaxies. The left plot visualizes a credible interval from our method, compared with 90% confidence intervals around the classical and PPI estimator. The orange and red bars display confidence intervals around the classical estimator when all imputed data are treated as real, where the imputation is done by thresholding and by sampling respectively. The right plot displays the width of credible/confidence intervals as a function of the labeled training data size $n$.
• Figure 3: Frequentist coverage of AI posterior 90% credible intervals for the median expression level of a gene induced by a promoter sequence. Intervals are from the 0.05 to 0.95 posterior quantile, using $n=2000$ samples in the analysis, and the AI prior described in Section \ref{['sec:additional-experiments']}. We display the actual coverage computed using oracle knowledge, and a bootstrapped estimated of the coverage computable via sample information only. Vertical lines denote possible choices of $\alpha$.
• Figure 4: Interval width and coverage of AI posterior credible intervals on experiments from angelopoulos2023prediction. Left: size of the AI posterior credible interval for specific $\alpha$ choices as a function of $n$. Middle: interval width as a function of $\alpha$ for the largest $n$. Right: empirical coverage of the intervals as a function of $\alpha$ for the largest $n$.

Theorems & Definitions (2)

• Theorem 1
• proof