Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantization Of Probability Measures In Maximum~Mean~Discrepancy Distance

Zahra Mehraban, Alois Pichler

TL;DR

The paper tackles quantizing probability measures by minimizing the maximum mean discrepancy (MMD) in an RKHS, proposing a two-stage approach that first determines optimal weights for a fixed set of support points and then optimizes the locations. By reformulating the objective on a product space, it enables stochastic gradient methods, while also delivering deterministic solutions for Gaussian kernels and explicit weight formulas for probability measures. Key contributions include explicit optimal weights for general and probability measures, a stochastic-location optimization framework with a cost function defined over pairs drawn from the data distribution, handling of non-negativity constraints, and thorough numerical validation across several distributions with kernel choices such as Gaussian and Matérn. The results yield scalable, accurate quantization of continuous distributions, with direct implications for machine learning and signal processing tasks where tractable finite representations of measures are crucial.

Abstract

Accurate approximation of probability measures is essential in numerical applications. This paper explores the quantization of probability measures using the maximum mean discrepancy (MMD) distance as a guiding metric. We first investigate optimal approximations by determining the best weights, followed by addressing the problem of optimal facility locations. To facilitate efficient computation, we reformulate the nonlinear objective as expectations over a product space, enabling the use of stochastic approximation methods. For the Gaussian kernel, we derive closed-form expressions to develop a deterministic optimization approach. By integrating stochastic approximation with deterministic techniques, our framework achieves precise and efficient quantization of continuous distributions, with significant implications for machine learning and signal processing applications.

Quantization Of Probability Measures In Maximum~Mean~Discrepancy Distance

TL;DR

The paper tackles quantizing probability measures by minimizing the maximum mean discrepancy (MMD) in an RKHS, proposing a two-stage approach that first determines optimal weights for a fixed set of support points and then optimizes the locations. By reformulating the objective on a product space, it enables stochastic gradient methods, while also delivering deterministic solutions for Gaussian kernels and explicit weight formulas for probability measures. Key contributions include explicit optimal weights for general and probability measures, a stochastic-location optimization framework with a cost function defined over pairs drawn from the data distribution, handling of non-negativity constraints, and thorough numerical validation across several distributions with kernel choices such as Gaussian and Matérn. The results yield scalable, accurate quantization of continuous distributions, with direct implications for machine learning and signal processing tasks where tractable finite representations of measures are crucial.

Abstract

Accurate approximation of probability measures is essential in numerical applications. This paper explores the quantization of probability measures using the maximum mean discrepancy (MMD) distance as a guiding metric. We first investigate optimal approximations by determining the best weights, followed by addressing the problem of optimal facility locations. To facilitate efficient computation, we reformulate the nonlinear objective as expectations over a product space, enabling the use of stochastic approximation methods. For the Gaussian kernel, we derive closed-form expressions to develop a deterministic optimization approach. By integrating stochastic approximation with deterministic techniques, our framework achieves precise and efficient quantization of continuous distributions, with significant implications for machine learning and signal processing applications.

Paper Structure

This paper contains 13 sections, 6 theorems, 84 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Outline of the paper.
  3. Mathematical setting
  4. The quantization weights
  5. Approximations of general measures
  6. Approximation of probability measures
  7. Optimal locations
  8. Stochastic optimization
  9. Stochastic gradient
  10. Non-negativity constraints
  11. Quantization of the normal distribution
  12. Numerical illustration
  13. Summary

Key Result

Proposition 1

The optimal weights $\bm \mu= [\mu_1,\dots,\mu_n]$ in eq:5 are given by where $\bm{K}$ is the matrix with entries and the entries of the vector $\bm{m}$ are evaluations of the embedding of the initial function at the support points, for $i=1,\dots,n$.

Figures (3)

  • Figure 1: Approximation of the normal distribution ($P$) with Matérn kernels with different bandwidth $\ell$ and smoothness $\nu$ with $n=5$ support points. Density of $P$ with probability mass function (left), and their embeddings $P_k,\ P_k^5\in \mathcal{H}_k$ (right); cf. also Table \ref{['tab:normal_distribution_results']}
  • Figure 2: Approximations of the uniform distribution, cf. also Table \ref{['tab:uniform_distribution_results']}
  • Figure 3: Quantization of the exponential distribution, cf. Table \ref{['tab:exponential_distribution_results']}

Theorems & Definitions (17)

  • Proposition 1
  • proof
  • Corollary 2
  • proof
  • Remark 3
  • Theorem 4
  • proof
  • Corollary 5
  • proof
  • Remark 6: Accumulated costs for involving probability measures
  • ...and 7 more