Normalizing Flow Regression for Bayesian Inference with Offline Likelihood Evaluations

TL;DR

Normalizing Flow Regression (NFR) provides an offline, surrogate-based approach to Bayesian posteriors by regressing log-density observations with a masked autoregressive flow and a learnable log-normalizing constant. Key innovations include a Tobit-inspired regression likelihood with noise shaping, informative priors over the flow, and an annealed optimization strategy that stabilizes training. Across synthetic and real-world problems, NFR often matches or surpasses baselines (Laplace, BBVI, VSBQ) in estimating posterior quantities and model evidence, while avoiding the need for new costly likelihood evaluations. The method offers a practical pathway for uncertainty quantification when standard Bayesian methods are prohibitive due to expensive model evaluations, demonstrated on domains from neuroscience to ecology and multisensory perception.

Abstract

Bayesian inference with computationally expensive likelihood evaluations remains a significant challenge in many scientific domains. We propose normalizing flow regression (NFR), a novel offline inference method for approximating posterior distributions. Unlike traditional surrogate approaches that require additional sampling or inference steps, NFR directly yields a tractable posterior approximation through regression on existing log-density evaluations. We introduce training techniques specifically for flow regression, such as tailored priors and likelihood functions, to achieve robust posterior and model evidence estimation. We demonstrate NFR's effectiveness on synthetic benchmarks and real-world applications from neuroscience and biology, showing superior or comparable performance to existing methods. NFR represents a promising approach for Bayesian inference when standard methods are computationally prohibitive or existing model evaluations can be recycled.

Figures (15)

• Figure 1: Illustration of the censoring effect of the Tobit likelihood on a target density. The left panel shows the density plot, while the right panel displays the corresponding log-density values. The shaded region represents the censored observations with log-density values below $y_{\text{low}}$, where the density is near-zero.
• Figure 2: Annealed optimization strategy. The flow regression model is progressively fitted to a series of tempered observations, with the inverse temperature $\beta$ increasing over multiple training iterations, interpolating between the base and unnormalized target distributions.
• Figure 3: Effect of prior variance on normalizing flow behavior, using a standard Gaussian as the base distribution. The panels show flow realizations with different prior standard deviations $\sigma_{{\bm{\phi}}}$: (a) The flow closely resembles the base distribution. (b) The flow exhibits controlled flexibility, allowing meaningful deviations while maintaining reasonable shapes. (c) The flow deviates significantly, producing complex and less plausible distributions.
• Figure 4: Multivariate Rosenbrock-Gaussian ($D=6$). Example contours of the marginal density for $x_3$ and $x_4$, for different methods. Ground-truth samples are in gray.
• Figure A.1: Diagnostics using corner plots. The orange density contours represent the flow posterior samples, while the blue points indicate training data for flow regression. (a) The flow’s probability mass escapes into regions with few or no training points, highlighting an unreliable flow approximation. (b) The high-probability region of the flow is well supported by training points, indicating that the qualitative diagnostic check is passed.