Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orthogonal Polynomials Approximation Algorithm (OPAA):a functional analytic approach to estimating probability densities

Lilian W. Bialokozowicz

TL;DR

OPAA tackles the challenge of estimating probability densities and their normalizing weights without relying on priors or independence assumptions. It introduces a functional-analytic framework that represents the square root of the density in an $L^2$ space using a complete Hermite basis, so that the density norm equals the sum of squared transform coefficients and the evidence can be obtained from these coefficients. The method relies on Gauss–Hermite quadrature to compute the expansion coefficients in a parallelizable, one-pass scheme, with universal factors that depend only on the quadrature order and basis degree. The key contributions are the functional perspective, a computation scheme that avoids sampling variance, and explicit mechanisms for scalable high-dimensional density estimation and Bayesian evidence evaluation. Together, these enable accurate, efficient density estimation and posterior normalization in complex Bayesian problems.

Abstract

We present the new Orthogonal Polynomials Approximation Algorithm (OPAA), a parallelizable algorithm that estimates probability distributions using functional analytic approach: first, it finds a smooth functional estimate of the probability distribution, whether it is normalized or not; second, the algorithm provides an estimate of the normalizing weight; and third, the algorithm proposes a new computation scheme to compute such estimates. A core component of OPAA is a special transform of the square root of the joint distribution into a special functional space of our construct. Through this transform, the evidence is equated with the $L^2$ norm of the transformed function, squared. Hence, the evidence can be estimated by the sum of squares of the transform coefficients. Computations can be parallelized and completed in one pass. OPAA can be applied broadly to the estimation of probability density functions. In Bayesian problems, it can be applied to estimating the normalizing weight of the posterior, which is also known as the evidence, serving as an alternative to existing optimization-based methods.

Orthogonal Polynomials Approximation Algorithm (OPAA):a functional analytic approach to estimating probability densities

TL;DR

OPAA tackles the challenge of estimating probability densities and their normalizing weights without relying on priors or independence assumptions. It introduces a functional-analytic framework that represents the square root of the density in an space using a complete Hermite basis, so that the density norm equals the sum of squared transform coefficients and the evidence can be obtained from these coefficients. The method relies on Gauss–Hermite quadrature to compute the expansion coefficients in a parallelizable, one-pass scheme, with universal factors that depend only on the quadrature order and basis degree. The key contributions are the functional perspective, a computation scheme that avoids sampling variance, and explicit mechanisms for scalable high-dimensional density estimation and Bayesian evidence evaluation. Together, these enable accurate, efficient density estimation and posterior normalization in complex Bayesian problems.

Abstract

We present the new Orthogonal Polynomials Approximation Algorithm (OPAA), a parallelizable algorithm that estimates probability distributions using functional analytic approach: first, it finds a smooth functional estimate of the probability distribution, whether it is normalized or not; second, the algorithm provides an estimate of the normalizing weight; and third, the algorithm proposes a new computation scheme to compute such estimates. A core component of OPAA is a special transform of the square root of the joint distribution into a special functional space of our construct. Through this transform, the evidence is equated with the norm of the transformed function, squared. Hence, the evidence can be estimated by the sum of squares of the transform coefficients. Computations can be parallelized and completed in one pass. OPAA can be applied broadly to the estimation of probability density functions. In Bayesian problems, it can be applied to estimating the normalizing weight of the posterior, which is also known as the evidence, serving as an alternative to existing optimization-based methods.
Paper Structure (19 sections, 3 theorems, 62 equations, 2 figures, 1 algorithm)

This paper contains 19 sections, 3 theorems, 62 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Polynomials in Machine Learning
  3. Structure of the Paper
  4. The Orthogonal Polynomials Approximation Algorithm (OPAA)
  5. Problem Statement
  6. Outline of the Algorithm
  7. Outline of the Computation Scheme
  8. Our Contributions
  9. Supporting Theorems in Functional Analysis
  10. Building Blocks of the Algorithm
  11. Hermite Polynomials and Density of Polynomials
  12. Gauss--Hermite Quadrature in One Dimension
  13. Quadrature In Higher Dimensions
  14. Vectorization and Parallelization
  15. Conclusions
  16. ...and 4 more sections

Key Result

Lemma 3.1

Every probability density $f(\theta)$ on $\mathbb{R}$ (that is, $f(\theta) \geq 0$ and $\int f(\theta) d\theta=1$) can be estimated as follows where $h_j(\theta)$ are Hermite functions of degree $j$.

Figures (2)

  • Figure 1: Plots of Hermite functions for $n=2, 5, 10$.
  • Figure 2: Example of a Mixed Gaussian Model.

Theorems & Definitions (3)

  • Lemma 3.1
  • Theorem 4.1: Density of polynomials in $L^2$
  • Theorem 4.2