Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tensor Density Estimator by Convolution-Deconvolution

Yifan Peng, Siyao Yang, Yuehaw Khoo, Daren Wang

TL;DR

This work tackles high-dimensional density estimation by introducing a convolution–deconvolution framework that produces a tensor-train representation of the density. The core idea is to express the target density $p^*(x)$ as a perturbation of a mean-field density $ ext{mean-field } oldsymbol{ u}(x)$ and project the empirical distribution onto a low-order cluster basis, followed by TT-compression (TT-SVD and its variants) and a deconvolution step to recover the density. The authors present several TT-compression algorithms (TT-SVD, TT-SVD-kn, TT-SVD-c, TT-rSVD-t), with theoretical error bounds showing linear scaling in dimension under mild assumptions, and validate the approach on Gaussian mixtures, Ginzburg–Landau Boltzmann densities, and MNIST, achieving competitive accuracy and scalability relative to neural methods. The results suggest a practical, linear-time alternative to deep generative models for high-dimensional density estimation, with robust variance control and efficient evaluations in TT form.

Abstract

We propose a linear algebraic framework for performing density estimation. It consists of three simple steps: convolving the empirical distribution with certain smoothing kernels to remove the exponentially large variance; compressing the empirical distribution after convolution as a tensor train, with efficient tensor decomposition algorithms; and finally, applying a deconvolution step to recover the estimated density from such tensor-train representation. Numerical results demonstrate the high accuracy and efficiency of the proposed methods.

Tensor Density Estimator by Convolution-Deconvolution

TL;DR

This work tackles high-dimensional density estimation by introducing a convolution–deconvolution framework that produces a tensor-train representation of the density. The core idea is to express the target density as a perturbation of a mean-field density and project the empirical distribution onto a low-order cluster basis, followed by TT-compression (TT-SVD and its variants) and a deconvolution step to recover the density. The authors present several TT-compression algorithms (TT-SVD, TT-SVD-kn, TT-SVD-c, TT-rSVD-t), with theoretical error bounds showing linear scaling in dimension under mild assumptions, and validate the approach on Gaussian mixtures, Ginzburg–Landau Boltzmann densities, and MNIST, achieving competitive accuracy and scalability relative to neural methods. The results suggest a practical, linear-time alternative to deep generative models for high-dimensional density estimation, with robust variance control and efficient evaluations in TT form.

Abstract

We propose a linear algebraic framework for performing density estimation. It consists of three simple steps: convolving the empirical distribution with certain smoothing kernels to remove the exponentially large variance; compressing the empirical distribution after convolution as a tensor train, with efficient tensor decomposition algorithms; and finally, applying a deconvolution step to recover the estimated density from such tensor-train representation. Numerical results demonstrate the high accuracy and efficiency of the proposed methods.

Paper Structure

This paper contains 42 sections, 31 theorems, 214 equations, 19 figures, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Main idea
  4. Related work
  5. Organization
  6. Tensor notations and diagrams
  7. Main approach
  8. Convolution step
  9. Deconvolution step
  10. Motivation: mean-field example
  11. Tensor-train SVD (TT-SVD)
  12. Fast implementation of TT-SVD
  13. TT-SVD with kernel Nyström approximation (TT-SVD-kn)
  14. TT-SVD with truncated cluster basis (TT-SVD-c)
  15. Acceleration of TT-SVD-c via hierarchical matrix decomposition
  16. ...and 27 more sections

Key Result

Lemma 1

Suppose $\{\phi_l\}_{l=1}^n$ are Legendre polynomials and the ground truth density $p^*$ satisfies 2-cluster_recover with the mean-field component $\mu \sim \emph{Unif}([0,1]^d)$, the following variance estimation holds

Figures (19)

  • Figure 1: Density estimation for data sampled from a Boltzmann distribution. Figure visualizes the second-moment error for different dimensionality $d$. We compare the result of one of our proposed tensor-train-based method called TT-SVD-kn (see Section \ref{['sec:tt-svd-kn']}) with that of one neural network approach (Masked Autoregressive Flow).
  • Figure 2: Basic tensor diagrams and operations.
  • Figure 3: Diagram for tensor-train representation. Left: discrete case. Right: continuous case.
  • Figure 4: Tensor diagram of $\hat{c}= \text{diag}(\tilde{w}_{\alpha})\Phi^T \hat{p}$. "$\alpha$" denotes diagonal matrix $\text{diag}(1,\alpha,\cdots,\alpha)\in \mathbb{R}^{n\times n}$; "$\phi_j$" denotes "matrix" $[\phi^j_l(x)]_{x,l}$.
  • Figure 5: Tensor diagram for the final density estimator $\tilde{p}$. "$\alpha^{-1}$" denotes diagonal matrix $\text{diag}(1,1/\alpha,\cdots,1/\alpha)\in \mathbb{R}^{n\times n}$; "$\phi_j$" denotes "matrix" $[\phi^j_l(x)]_{x,l}$.
  • ...and 14 more figures

Theorems & Definitions (65)

  • Lemma 1
  • proof
  • Remark 1
  • Theorem 1
  • Corollary 1
  • Definition 1
  • Definition 2
  • proof
  • proof
  • Lemma 2
  • ...and 55 more