Fourier Basis Density Model

TL;DR

The paper addresses univariate density estimation for challenging multi-modal distributions by introducing the Fourier basis density model (FBM), which uses a truncated Fourier series with coefficients constrained by $c_n = \sum_{k=0}^{N-n} a_k a_{k+n}^*$ to guarantee non-negativity via Herglotz's theorem and a simple normalization with $Z = 2 c_0$. It extends the periodic representation to the real line through a $g(x; s, t) = s \tanh^{-1}(x) + t$ mapping, yielding a density $q(x; θ, s, t)$ with a tractable Jacobian. The approach demonstrates parameter efficiency and competitive cross-entropy performance against a deep factorized baseline across multi-modal densities, and shows promise as an entropy model for rate-distortion optimization in learned compression as evidenced by banana-distribution experiments. Overall, the method provides a transparent, frequency-based bias toward smooth densities and a practical, end-to-end trainable alternative for univariate density estimation and compression tasks.

Abstract

We introduce a lightweight, flexible and end-to-end trainable probability density model parameterized by a constrained Fourier basis. We assess its performance at approximating a range of multi-modal 1D densities, which are generally difficult to fit. In comparison to the deep factorized model introduced in [1], our model achieves a lower cross entropy at a similar computational budget. In addition, we also evaluate our method on a toy compression task, demonstrating its utility in learned compression.

Figures (9)

• Figure 1: Model fit for mixture of beta distribution. a) Density plot for a $64$ term Fourier basis density model (best viewed on screen). b) The fit improves with increasing number of parameters.
• Figure 2: Model fit for mixture of logit-normals distribution.
• Figure 3: Model fit for mixture of 3 Gaussians. a) Density plot for a $64$ term Fourier basis density model (best viewed on screen). b) The fit improves with increasing number of parameters.
• Figure 4: Model fit for mixture of Gaussians and Laplacians.
• Figure 5: a) Model fit for mixture of $K$ Gaussians (best viewed on screen). b) KLD between model and target as a function of $K$, for deep factorized model (DFP) and Fourier basis density model (FBM) on a fixed parameter budget ($\sim 90$ parameters).