Convolve and Conquer: Data Comparison with Wiener Filters

TL;DR

The paper tackles the limitations of mean-squared-error in capturing distributional structure for data comparison by introducing a Wiener-filter-based framework that enforces global, convolutional similarity between samples. It defines the Wiener loss, which preserves amplitude information and aligns samples through full-lag Wiener filters, and extends to a Wiener diffusion energy-based model for non-parametric generation via Langevin dynamics. Across autoencoding, medical-imaging imputation, generative modelling, and translation-invariant classification, the approach yields improved perceptual quality and robustness to translations, while enabling a novel non-parametric generative mechanism. The work also discusses computational considerations, hyperparameter choices, and potential for broad applicability in tasks requiring preservation of global data correlations.

Abstract

Quantitative evaluations of differences and/or similarities between data samples define and shape optimisation problems associated with learning data distributions. Current methods to compare data often suffer from limitations in capturing such distributions or lack desirable mathematical properties for optimisation (e.g. smoothness, differentiability, or convexity). In this paper, we introduce a new method to measure (dis)similarities between paired samples inspired by Wiener-filter theory. The convolutional nature of Wiener filters allows us to comprehensively compare data samples in a globally correlated way. We validate our approach in four machine learning applications: data compression, medical imaging imputation, translated classification, and non-parametric generative modelling. Our results demonstrate increased resolution in reconstructed images with better perceptual quality and higher data fidelity, as well as robustness against translations, compared to conventional mean-squared-error analogue implementations.

Figures (5)

• Figure 1: Top: visual illustration of two-dimensional Wiener filters that match MNIST sample $\mathbf{y}$ to random noise $\mathbf{x}$. Middle: matching $\mathbf{y}$ to $\mathbf{x}$ where $\mathbf{x}=\mathbf{y}$, demonstrating that the Wiener filter when $\mathbf{x}=\mathbf{y}$ is given by the Dirac delta function. Bottom: matching $\mathbf{y}$ to $\mathbf{x}$ where $\mathbf{x}$ is a translation of $\mathbf{y}$.
• Figure 2: Autoencoder reconstruction outputs from the validation dataset. Comparison of Ground Truth (top row), mean squared-error (middle row), and Wiener loss (bottom row) as the optimisation functional.
• Figure 3: Ground truth (a-b) and zoomed outputs from imputing under-sampled scans (c) using bicubic interpolation (d), U-net ronneberger2015u with MSE (e) and U-net ronneberger2015u with Wiener loss (f). The Wiener loss is able to recover finer structures of the scan that are aliased or blurred in other methods. Results of (e-f) are obtained from the validation set.
• Figure 4: Latent samples generated using the Wiener diffusion, visualised using the decoder, and an illustration of the diffusion process alongside the 30 closest Wiener filters to the sample $\mathbf{x}_{t}$ (as defined by \ref{['eqn:wiener-diffusion-energy']}) for MNIST (top) and CelebA (bottom).
• Figure 5: Image classification task to use the MNIST training data (left) to predict the affNIST testing data (right). A KNN classifier with the Manhattan distance metric achieves and accuracy of $\mathbf{14}\boldsymbol{\%}$, whereas a KNN classifier with our proposed translation invariant metric $\mathcal{L}_{\textsc{ti}}$ achieves an accuracy of $\mathbf{61}\boldsymbol{\%}$