Fast Bayesian Updates via Harmonic Representations
Di Zhang
TL;DR
The paper tackles the computational bottleneck of Bayesian inference caused by the intractable marginal likelihood p(\mathcal{D}). It introduces a harmonic-analysis framework that expands the prior and likelihood in an orthonormal basis and shows that Bayesian updating becomes a spectral convolution: the unnormalized posterior coefficients satisfy \tilde{c}_k = (a*b)_k with Z = \tilde{c}_0 and p(\theta|\mathcal{D}) = \tilde{p}(\theta|\mathcal{D})/Z. By applying spectral truncation to a finite set of coefficients and periodizing, the update reduces to a circular convolution, enabling an FFT-based implementation with complexity $O(N \log N)$ per update. The method relies on smoothness and spectral-decay properties to guarantee accurate finite-dimensional approximations, and it offers real-time, deterministic sequential inference while bridging Bayesian computation with signal processing concepts. Limitations arise for discontinuous or heavy-tailed priors and high-dimensional problems, but the framework points to adaptive bases, tensor decompositions, and extensions to sequential filtering and nonparametric models as promising directions.
Abstract
Bayesian inference, while foundational to probabilistic reasoning, is often hampered by the computational intractability of posterior distributions, particularly through the challenging evidence integral. Conventional approaches like Markov Chain Monte Carlo (MCMC) and Variational Inference (VI) face significant scalability and efficiency limitations. This paper introduces a novel, unifying framework for fast Bayesian updates by leveraging harmonic analysis. We demonstrate that representing the prior and likelihood in a suitable orthogonal basis transforms the Bayesian update rule into a spectral convolution. Specifically, the Fourier coefficients of the posterior are shown to be the normalized convolution of the prior and likelihood coefficients. To achieve computational feasibility, we introduce a spectral truncation scheme, which, for smooth functions, yields an exceptionally accurate finite-dimensional approximation and reduces the update to a circular convolution. This formulation allows us to exploit the Fast Fourier Transform (FFT), resulting in a deterministic algorithm with O(N log N) complexity -- a substantial improvement over the O(N^2) cost of naive methods. We establish rigorous mathematical criteria for the applicability of our method, linking its efficiency to the smoothness and spectral decay of the involved distributions. The presented work offers a paradigm shift, connecting Bayesian computation to signal processing and opening avenues for real-time, sequential inference in a wide class of problems.