Explaining the Success of Nearest Neighbor Methods in Prediction

TL;DR

This work synthesizes theory and practice around nearest neighbor prediction by deriving nonasymptotic guarantees for k-NN, fixed-radius NN, and kernel regression/classification in general metric spaces, anchored by smoothness or Besicovitch-type conditions. It shows how clustering structure enables reliable prediction in diverse tasks like time series forecasting, online collaborative filtering, and patch-based image segmentation, and it connects these guarantees to practical approximate NN methods and distance learning. The monograph also details plug-in classification via regression estimators, explores adaptive neighbor and bandwidth selection, and presents three contemporary applications with concrete nonasymptotic bounds and empirical validation. Overall, it provides a cohesive framework linking theory, scalable computation, and interpretability of NN-based prediction across domains.

Abstract

Many modern methods for prediction leverage nearest neighbor search to find past training examples most similar to a test example, an idea that dates back in text to at least the 11th century and has stood the test of time. This monograph aims to explain the success of these methods, both in theory, for which we cover foundational nonasymptotic statistical guarantees on nearest-neighbor-based regression and classification, and in practice, for which we gather prominent methods for approximate nearest neighbor search that have been essential to scaling prediction systems reliant on nearest neighbor analysis to handle massive datasets. Furthermore, we discuss connections to learning distances for use with nearest neighbor methods, including how random decision trees and ensemble methods learn nearest neighbor structure, as well as recent developments in crowdsourcing and graphons. In terms of theory, our focus is on nonasymptotic statistical guarantees, which we state in the form of how many training data and what algorithm parameters ensure that a nearest neighbor prediction method achieves a user-specified error tolerance. We begin with the most general of such results for nearest neighbor and related kernel regression and classification in general metric spaces. In such settings in which we assume very little structure, what enables successful prediction is smoothness in the function being estimated for regression, and a low probability of landing near the decision boundary for classification. In practice, these conditions could be difficult to verify for a real dataset. We then cover recent guarantees on nearest neighbor prediction in the three case studies of time series forecasting, recommending products to people over time, and delineating human organs in medical images by looking at image patches. In these case studies, clustering structure enables successful prediction.

Figures (19)

• Figure 1: Illustration to help with $k$-NN analysis ($k=6$ in this example): the blue points are training data, the test feature vector that we are making a prediction for is the black point $x$ and its $k$-th nearest neighbor $X_{(k+1)}(x)$ is on the boundary of the shaded ball, which has radius $\rho(x, X_{(k+1)}(x))$.
• Figure 2: Example where $k$-NN regression accuracy can be low ($k=6$ in this example): when the feature distribution $\mathbb{P}_X$ is univariate Gaussian, training data (blue points) are likely to land near the mean of the Gaussian, e.g., mostly within the green region labeled $\mathcal{X}_{\text{good}}$. If we want to estimate $\eta(x)$ for $x$ very far from the mean, then it is likely that its $k$ nearest neighbors (circled in orange) are not close to $x$, and unless $\eta$ is extremely smooth, then regression estimate $\widehat{\eta}_{k\text{-NN}}(x)$ will be inaccurate.
• Figure 3: Diagram to help explain the alternative strategy for guaranteeing low expected regression error. The shaded region is the feature space $\mathcal{X}$, which is covered by small balls each with radius $h^*/2$. Test point $x$ lands in one of these small balls. Regardless of where $x$ lands, the small ball that it lands in is contained in the ball with radius $h^*$ centered at $x$. Enough training data should be collected so nearly every small ball has at least $k$ training points land in it.
• Figure 4: Illustration to help with fixed-radius NN analysis: the blue points are training data, the test feature vector that we are making a prediction for is the black point $x$ and the nearest neighbors used are the ones inside the shaded ball, which has radius $h$.
• Figure 5: Probability density function of $\mathbb{P}_X\sim\text{Uniform}[a,b]$, and examples of two different balls that a feature vector drawn from $\mathbb{P}_X$ can land in, one ball completely contained in $[a,b]$ (top), and one ball on an endpoint of $[a,b]$ (bottom).

Theorems & Definitions (12)

• proof : Proof of Proposition \ref{['prop:regression-function-minimizes-expected-square-error']}
• proof : Proof of Proposition \ref{['prop:bayes-classifier-minimizes-prob-error']}
• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof