Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models

TL;DR

The paper addresses the challenge of expensive simulations in simulation-based inference by repurposing TabPFN, a tabular foundation model, as a training-free autoregressive density estimator for Bayesian posterior inference (NPE-PFN). By modeling the posterior factors autoregressively and employing filtering to extend the effective context, the approach achieves competitive accuracy with substantially fewer simulations, and a sequential variant (TSNPE-PFN) achieves strong performance with limited budgets. The method exhibits robustness to misspecification and scales to complex, high-dimensional problems like Hodgkin-Huxley and pyloric network models, where it markedly outperforms traditional SBI methods in simulation efficiency. Overall, NPE-PFN offers a simple, training-free, and flexible SBI workflow that lowers barriers for non-experts and enables efficient inference on expensive simulators.

Abstract

Simulation-based inference (SBI) offers a flexible and general approach to performing Bayesian inference: In SBI, a neural network is trained on synthetic data simulated from a model and used to rapidly infer posterior distributions for observed data. A key goal for SBI is to achieve accurate inference with as few simulations as possible, especially for expensive simulators. In this work, we address this challenge by repurposing recent probabilistic foundation models for tabular data: We show how tabular foundation models -- specifically TabPFN -- can be used as pre-trained autoregressive conditional density estimators for SBI. We propose Neural Posterior Estimation with Prior-data Fitted Networks (NPE-PFN) and show that it is competitive with current SBI approaches in terms of accuracy for both benchmark tasks and two complex scientific inverse problems. Crucially, it often substantially outperforms them in terms of simulation efficiency, sometimes requiring orders of magnitude fewer simulations. NPE-PFN eliminates the need for inference network selection, training, and hyperparameter tuning. We also show that it exhibits superior robustness to model misspecification and can be scaled to simulation budgets that exceed the context size limit of TabPFN. NPE-PFN provides a new direction for SBI, where training-free, general-purpose inference models offer efficient, easy-to-use, and flexible solutions for a wide range of stochastic inverse problems.

Figures (18)

• Figure 1: Comparison of NPE to NPE-PFN (ours): Both approaches use simulations sampled from the prior and simulator. In (standard) NPE, a neural density estimator is trained to obtain the posterior. In NPE-PFN, the posterior is evaluated by autoregressively passing the simulation dataset and observations to TabPFN.
• Figure 2: SBI benchmark results for amortized and sequential NPE-PFN.(a) C2ST for NPE, NLE, and NPE-PFN across ten reference posteriors (lower is better); dots indicate averages and bars show 95% confidence intervals over five independent runs. (b) Average time to generate $10^4$ posterior samples and (if applicable) training time. (c) C2ST for sequential methods.
• Figure 3: Robustness under misspecification. C2ST against a well-specified ground truth posterior. The red star marks the well-specified model. Rows: Prior and likelihood misspecification. Columns: NPE, NPE-PFN, and their difference in terms of C2ST accuracy (darker blue indicates that NPE-PFN is better).
• Figure 4: Posterior inference for observations from the Allen cell type database.(a) Average distance to observation in standardized space of summary statistics for both the real and synthetic observations. (b) Posterior predictive simulations for real observations using TSNPE-PFN and TSNPE. (c) Posterior marginals for one synthetic observation; TSNPE-PFN marginals are substantially more constrained for several parameters. (d) Simulation-based calibration for NPE-PFN and NPE on the HH simulator.
• Figure 5: Results on the pyloric simulator.(a) Voltage traces from the experimental measurement (top) and a posterior predictive simulated using the posterior mean from TSNPE-PFN as the parameter (bottom). (b) Average distance (energy scoring rule) to observation and percentage of valid simulation from posterior samples; compared to experimental results obtained in glaser2023maximumlikelihoodlearningunnormalized.