Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Random Wavelet Features: Efficient Non-Stationary Kernel Approximation with Convergence Guarantees

Sawan Kumar, Souvik Chakraborty

TL;DR

The paper tackles the challenge of modeling non-stationary processes at scale by replacing stationary random Fourier features with Random Wavelet Features (RWF), which sample from localized wavelet families to form explicit, multi-resolution feature maps. RWF yields a non-stationary kernel with positive definiteness, unbiasedness, and uniform convergence guarantees, while maintaining training complexity of $\mathcal{O}(ND^2)$ and test-time cost $\mathcal{O}(D^2)$. Theoretical results include bounds on variance, a probabilistic uniform convergence bound, and explicit sample complexity for achieving a given approximation error. Empirically, RWF-GP outperforms stationary baselines and remains competitive with more complex non-stationary models across synthetic, speech, and large real-world datasets, highlighting the method’s practicality for scalable, expressive kernel learning.

Abstract

Modeling non-stationary processes, where statistical properties vary across the input domain, is a critical challenge in machine learning; yet most scalable methods rely on a simplifying assumption of stationarity. This forces a difficult trade-off: use expressive but computationally demanding models like Deep Gaussian Processes, or scalable but limited methods like Random Fourier Features (RFF). We close this gap by introducing Random Wavelet Features (RWF), a framework that constructs scalable, non-stationary kernel approximations by sampling from wavelet families. By harnessing the inherent localization and multi-resolution structure of wavelets, RWF generates an explicit feature map that captures complex, input-dependent patterns. Our framework provides a principled way to generalize RFF to the non-stationary setting and comes with a comprehensive theoretical analysis, including positive definiteness, unbiasedness, and uniform convergence guarantees. We demonstrate empirically on a range of challenging synthetic and real-world datasets that RWF outperforms stationary random features and offers a compelling accuracy-efficiency trade-off against more complex models, unlocking scalable and expressive kernel methods for a broad class of real-world non-stationary problems.

Scalable Random Wavelet Features: Efficient Non-Stationary Kernel Approximation with Convergence Guarantees

TL;DR

The paper tackles the challenge of modeling non-stationary processes at scale by replacing stationary random Fourier features with Random Wavelet Features (RWF), which sample from localized wavelet families to form explicit, multi-resolution feature maps. RWF yields a non-stationary kernel with positive definiteness, unbiasedness, and uniform convergence guarantees, while maintaining training complexity of and test-time cost . Theoretical results include bounds on variance, a probabilistic uniform convergence bound, and explicit sample complexity for achieving a given approximation error. Empirically, RWF-GP outperforms stationary baselines and remains competitive with more complex non-stationary models across synthetic, speech, and large real-world datasets, highlighting the method’s practicality for scalable, expressive kernel learning.

Abstract

Modeling non-stationary processes, where statistical properties vary across the input domain, is a critical challenge in machine learning; yet most scalable methods rely on a simplifying assumption of stationarity. This forces a difficult trade-off: use expressive but computationally demanding models like Deep Gaussian Processes, or scalable but limited methods like Random Fourier Features (RFF). We close this gap by introducing Random Wavelet Features (RWF), a framework that constructs scalable, non-stationary kernel approximations by sampling from wavelet families. By harnessing the inherent localization and multi-resolution structure of wavelets, RWF generates an explicit feature map that captures complex, input-dependent patterns. Our framework provides a principled way to generalize RFF to the non-stationary setting and comes with a comprehensive theoretical analysis, including positive definiteness, unbiasedness, and uniform convergence guarantees. We demonstrate empirically on a range of challenging synthetic and real-world datasets that RWF outperforms stationary random features and offers a compelling accuracy-efficiency trade-off against more complex models, unlocking scalable and expressive kernel methods for a broad class of real-world non-stationary problems.
Paper Structure (57 sections, 9 theorems, 59 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 57 sections, 9 theorems, 59 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries and Background
  4. Gaussian Processes
  5. Random Fourier Feature GPs (RFF-GP)
  6. Proposed methodology
  7. Wavelet-based Kernel Construction
  8. Random Wavelet Feature Sampling Strategy
  9. Practical Considerations
  10. Theoretical Analysis
  11. Positive Definiteness of Wavelet Kernels
  12. Unbiasedness and Variance Bounds
  13. Uniform Convergence Guarantees
  14. Sample Complexity Analysis
  15. Experiments
  16. ...and 42 more sections

Key Result

Theorem 4.1

Let $\psi: \mathbb{R}^d \to \mathbb{R}$ be a mother wavelet function, and define the family of wavelets as $\psi_{s,t}(\bm{x}) = s^{-d/2} \psi\left( s^{-1} (\bm{x} - t) \right)$ for scale $s > 0$ and translation $t \in \mathbb{R}^d$. Let $p(s, t): \mathbb{R}^+ \times \mathbb{R}^d \to [0, \infty)$ be is a positive definite kernel on $\mathbb{R}^d \times \mathbb{R}^d$.

Figures (3)

  • Figure 1: Predictive performance of different GP methods on a step function regression task. Each panel shows the predictive mean (solid line) with $\pm 2\sigma$ confidence intervals (shaded), training data (dots). RWF-GP (ours) captures the discontinuities sharply while maintaining calibrated uncertainty. In contrast, Exact GP, Sparse Variational GP, and RFF-GP struggle with sharp transitions, either oversmoothing or miscalibrating the uncertainty.
  • Figure 2: Scalability on the multi-step function. Time and memory vs. number of training samples on the multi-step function: RWF is most efficient; SVGP and Deep GP incur higher cost.
  • Figure 3: RMSE vs. number of features $D$ for RWF-GP (Mexican-hat) and RFF-GP on the multi-step function.

Theorems & Definitions (13)

  • Definition 3.1: Random Wavelet Features
  • Theorem 4.1: Positive Definiteness of Wavelet-Based Kernels
  • Lemma 4.1: Unbiasedness
  • Lemma 4.2: Variance Bound
  • Theorem 4.2: Uniform Convergence of Random Wavelet Features
  • proof
  • proof
  • proof
  • Lemma A.1: Stationarity criterion vs. non-stationarity under bounded $p_t$
  • Lemma A.2: Wavelet localization: explicit feature bounds
  • ...and 3 more