Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A parametric algorithm is optimal for non-parametric regression of smooth functions

Davide Maran, Marcello Restelli

TL;DR

The paper addresses uniform $L^{\infty}$ regression for smooth functions $f:[-1,1]^d\to\mathbb{R}$ with active sampling. It introduces PADUA, a parametric algorithm based on Fourier-series projections and a De la Vallée-Poussin kernel, augmented by a convolution-based sampling trick to realize a misspecification-free target and enable simultaneous derivative estimation. The authors prove that PADUA achieves optimal (up to logarithmic factors) sample complexity and, in prediction, optimal space complexity, both in 1D and higher dimensions, and they establish matching lower bounds. Empirical validation on real audio data shows PADUA delivers comparable accuracy to state-of-the-art nonparametric methods while offering substantial speedups, demonstrating practical scalability for smooth-function regression tasks.

Abstract

We address the regression problem for a general function $f:[-1,1]^d\to \mathbb R$ when the learner selects the training points $\{x_i\}_{i=1}^n$ to achieve a uniform error bound across the entire domain. In this setting, known historically as nonparametric regression, we aim to establish a sample complexity bound that depends solely on the function's degree of smoothness. Assuming periodicity at the domain boundaries, we introduce PADUA, an algorithm that, with high probability, provides performance guarantees optimal up to constant or logarithmic factors across all problem parameters. Notably, PADUA is the first parametric algorithm with optimal sample complexity for this setting. Due to this feature, we prove that, differently from the non-parametric state of the art, PADUA enjoys optimal space complexity in the prediction phase. To validate these results, we perform numerical experiments over functions coming from real audio data, where PADUA shows comparable performance to state-of-the-art methods, while requiring only a fraction of the computational time.

A parametric algorithm is optimal for non-parametric regression of smooth functions

TL;DR

The paper addresses uniform regression for smooth functions with active sampling. It introduces PADUA, a parametric algorithm based on Fourier-series projections and a De la Vallée-Poussin kernel, augmented by a convolution-based sampling trick to realize a misspecification-free target and enable simultaneous derivative estimation. The authors prove that PADUA achieves optimal (up to logarithmic factors) sample complexity and, in prediction, optimal space complexity, both in 1D and higher dimensions, and they establish matching lower bounds. Empirical validation on real audio data shows PADUA delivers comparable accuracy to state-of-the-art nonparametric methods while offering substantial speedups, demonstrating practical scalability for smooth-function regression tasks.

Abstract

We address the regression problem for a general function when the learner selects the training points to achieve a uniform error bound across the entire domain. In this setting, known historically as nonparametric regression, we aim to establish a sample complexity bound that depends solely on the function's degree of smoothness. Assuming periodicity at the domain boundaries, we introduce PADUA, an algorithm that, with high probability, provides performance guarantees optimal up to constant or logarithmic factors across all problem parameters. Notably, PADUA is the first parametric algorithm with optimal sample complexity for this setting. Due to this feature, we prove that, differently from the non-parametric state of the art, PADUA enjoys optimal space complexity in the prediction phase. To validate these results, we perform numerical experiments over functions coming from real audio data, where PADUA shows comparable performance to state-of-the-art methods, while requiring only a fraction of the computational time.

Paper Structure

This paper contains 30 sections, 22 theorems, 98 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Simultaneous Approximation of the derivatives
  3. Paper structure
  4. Preliminaries
  5. Periodic functions and Fourier series
  6. Regression and Fourier Series
  7. The problem of linear regression under misspecification
  8. Fourier Series, Convolution, Kernels
  9. Fooling our own learner
  10. Algorithm and main results
  11. Multi-dimensional generalization
  12. Lower bound
  13. Related works
  14. Parametric approaches
  15. Non-Parametric approaches
  16. ...and 15 more sections

Key Result

Theorem 1

(Theorem 4.1 part (ii) from schultz1969multivariate) There exists an absolute constant $K>0$ such that, for any $f\in \mathcal{\mathcal{}} C_p^{\nu}([-1,1]^d)$ we have:

Figures (3)

  • Figure 1: Dirichlet Kernel (left) and de la Vallée-Poussin one (right) maran2024projection.
  • Figure 2: True unknown function $f(\cdot)$ used in the experiments and $95\%$ confidence regions for the predictions $f_n(\cdot)$ generated by three algorithms.
  • Figure 3: Left: $L^\infty$ error of each of the algorithm, each averaged over $5$ random seeds, and with shaded regions representing $95\%$ coverage confidence intervals for the estimation. Right: log-scale plot of the running time of each algorithm, compared to the number $n$ of samples.

Theorems & Definitions (35)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Corollary 4
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Theorem 7
  • proof
  • Theorem 8
  • ...and 25 more