Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Minimum discrepancy principle strategy for choosing $k$ in $k$-NN regression

Yaroslav Averyanov, Alain Celisse

TL;DR

Problem: selecting the hyperparameter $k$ in the $k$-NN regression estimator without relying on hold-out data. Approach: a data-driven early-stopping rule based on the minimum discrepancy principle, monitoring the empirical risk $R_k$ and stopping at $k^{\\tau}$ when $R_k \\le 2R_2$, thereby avoiding computation of all estimators. Contributions: non-asymptotic oracle-type bounds showing minimax-optimality over Lipschitz function classes on bounded domains, favorable empirical performance against Hold-out, 5-fold CV, and AIC, and reduced computational cost by not evaluating the full grid. Significance: provides a practical, scalable, data-driven method for hyperparameter selection in nonparametric regression with no hold-out data, supported by theory and experiments on artificial and real data.

Abstract

We present a novel data-driven strategy to choose the hyperparameter $k$ in the $k$-NN regression estimator without using any hold-out data. We treat the problem of choosing the hyperparameter as an iterative procedure (over $k$) and propose using an easily implemented in practice strategy based on the idea of early stopping and the minimum discrepancy principle. This model selection strategy is proven to be minimax-optimal over some smoothness function classes, for instance, the Lipschitz functions class on a bounded domain. The novel method often improves statistical performance on artificial and real-world data sets in comparison to other model selection strategies, such as the Hold-out method, 5-fold cross-validation, and AIC criterion. The novelty of the strategy comes from reducing the computational time of the model selection procedure while preserving the statistical (minimax) optimality of the resulting estimator. More precisely, given a sample of size $n$, if one should choose $k$ among $\left\{ 1, \ldots, n \right\}$, and $\left\{ f^1, \ldots, f^n \right\}$ are the estimators of the regression function, the minimum discrepancy principle requires the calculation of a fraction of the estimators, while this is not the case for the generalized cross-validation, Akaike's AIC criteria, or Lepskii principle.

Minimum discrepancy principle strategy for choosing $k$ in $k$-NN regression

TL;DR

Problem: selecting the hyperparameter in the -NN regression estimator without relying on hold-out data. Approach: a data-driven early-stopping rule based on the minimum discrepancy principle, monitoring the empirical risk and stopping at when , thereby avoiding computation of all estimators. Contributions: non-asymptotic oracle-type bounds showing minimax-optimality over Lipschitz function classes on bounded domains, favorable empirical performance against Hold-out, 5-fold CV, and AIC, and reduced computational cost by not evaluating the full grid. Significance: provides a practical, scalable, data-driven method for hyperparameter selection in nonparametric regression with no hold-out data, supported by theory and experiments on artificial and real data.

Abstract

We present a novel data-driven strategy to choose the hyperparameter in the -NN regression estimator without using any hold-out data. We treat the problem of choosing the hyperparameter as an iterative procedure (over ) and propose using an easily implemented in practice strategy based on the idea of early stopping and the minimum discrepancy principle. This model selection strategy is proven to be minimax-optimal over some smoothness function classes, for instance, the Lipschitz functions class on a bounded domain. The novel method often improves statistical performance on artificial and real-world data sets in comparison to other model selection strategies, such as the Hold-out method, 5-fold cross-validation, and AIC criterion. The novelty of the strategy comes from reducing the computational time of the model selection procedure while preserving the statistical (minimax) optimality of the resulting estimator. More precisely, given a sample of size , if one should choose among , and are the estimators of the regression function, the minimum discrepancy principle requires the calculation of a fraction of the estimators, while this is not the case for the generalized cross-validation, Akaike's AIC criteria, or Lepskii principle.

Paper Structure

This paper contains 30 sections, 12 theorems, 129 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Statistical model, main assumption, and notation
  3. $k$-NN estimator and minimum discrepancy stopping rule
  4. $k$-NN regression estimator
  5. Related work
  6. Bias-variance trade-off and minimum discrepancy principle rule
  7. Theoretical optimality result
  8. Empirical comparison to other model selection rules
  9. Description of the model selection rules to compare
  10. Artificial data
  11. Real data
  12. Conclusion
  13. Availability of data and material
  14. Code availability
  15. Competing interests
  16. ...and 15 more sections

Key Result

theorem 1

(Upper bound on the empirical norm). Under Assumption assumption_boundness, for arbitrary $v_1 \geq 0$ and $v_2 \geq \frac{4 \mathcal{M}^2}{\sigma^2}$, with probability at least $1 - 18 \exp \left(- v_1 \right) - 5\exp \left( - \min \left( 1, \frac{\sigma^2}{128 \mathcal{M}^2} \right) n v_2^2 \left( 1 - \frac{1}{n^{1.5}2^{n/2}} \right)^2 \right) - \frac{5 (v_2 + v_2^2)}{\sqrt{n \left( n/2 - 1 \ri

Figures (6)

  • Figure 1: Sq. bias, variance, risk and (expected) empirical risk behaviour.
  • Figure 2: $k$-NN estimator (\ref{['estimator']}) with two noised regression functions: smooth $f_1^*(x) = 1.5 \cdot \left[ \lVert x - 0.5 \rVert / \sqrt{3} - 0.5 \right]$ for panel (a) and "sinus" $f_2^*(x) = 1.5 \cdot \textnormal{sin}(\lVert x \rVert / \sqrt{3})$ for panel (b), with uniform covariates $x_j \overset{\textnormal{i.i.d.}}{\sim} \mathbb{U}[0, 1]^3$. Each curve corresponds to the $L_2(\mathbb{P}_n)$ squared norm error for the stopping rules (\ref{['k_tau']}), (\ref{['k_star']}), (\ref{['k_ho']}), (\ref{['k_1_out']}), averaged over $1000$ independent trials, versus the sample size $n = \{50, 80, 100, 160, 200, 250 \}$.
  • Figure 3: Stopping the learning process based on the rule (\ref{['k_tau']}) applied to two data sets: a) "Diabetes" and b) "Boston Housing Prices". "Threshold" horizontal line corresponds to the estimated variance equal to $2R_2$.
  • Figure 4: Runtime (in seconds) and $L_2(\mathbb{P}_n)$ prediction error versus sub-sample size for different model selection methods: MD principle (\ref{['k_tau']}), AIC (\ref{['k_aic']}), GCV (\ref{['k_1_out']}), and $5$--fold cross-validation (\ref{['k_vfcv']}), tested on the "Boston Housing Prices" and "Diabetes" data sets. In all cases, each point corresponds to the average of $25$ trials. (a), (c) Runtime verus the sub-sample size $n \in \{70, 88, 118, 177, 354 \}$. (b), (d) Least-squares prediction error $\lVert f^k - y_{\textnormal{test}} \rVert$ versus the sub-sample size $n \in \{70, 88, 118, 177, 354 \}$.
  • Figure 5: Runtime (in seconds) and $L_2(\mathbb{P}_n)$ prediction error versus sub-sample size for different model selection methods: MDP (\ref{['k_tau']}), AIC (\ref{['k_aic']}), GCV (\ref{['k_1_out']}), and $5$--fold cross-validation (\ref{['k_vfcv']}), tested on the "California Houses Prices" and "Power Plants" data set. In all cases, each point corresponds to the average of $25$ trials. (a), (c) Runtime verus the sub-sample size $n \in \{420, 525, 700, 1050, 2100 \}$. (b), (d) Least-squares prediction error $\lVert f^k - y_{\textnormal{test}} \rVert$ versus the sub-sample size $n \in \{420, 525, 700, 1050, 2100 \}$.
  • ...and 1 more figures

Theorems & Definitions (21)

  • theorem 1
  • proof
  • corollary 1
  • lemma 1: Concentration of a linear term
  • lemma 2: Hanson-Wright's inequality for Gaussian random variables in rudelson2013hanson
  • lemma 3: Hoeffding's inequality for bounded differences in wainwright2019high, p.454
  • lemma 4
  • proof
  • lemma 5
  • proof
  • ...and 11 more