Debiased Nonparametric Regression for Statistical Inference and Distributionally Robustness
Masahiro Kato
TL;DR
This work tackles the lack of statistical-inference guarantees for modern nonparametric regression by introducing a model-free debiasing approach that yields pointwise and uniform risk convergence and asymptotic normality under mild Hölder smoothness. The method proceeds in three stages: (i) estimate the target function $f_0$ with a smooth estimator $\widehat{f}_n$, (ii) estimate the conditional expected residual $\mathbb{E}[Y-f(X)\mid X]$ via local polynomial regression to obtain $\widehat{b}_n$, and (iii) form $\widetilde{f}_n=\widehat{f}_n+\widehat{b}_n$; this debiased estimator enjoys closed-form computation, bias-variance control, and double robustness. Key theoretical contributions show that, with $f_0-\widehat{f}_n\in\Sigma(s,L)$, the estimator achieves pointwise MSE rate $O(n^{-s/(2s+1)})$, $\,\sqrt{n h_n}\, (\widetilde{f}_n(x_0)-f_0(x_0))\to_d \mathcal{N}(0,V(x_0))$, and uniform convergence $\|\widetilde{f}_n-f_0\|_ty^2=O(((\log n)/n)^{2s/(2s+1)})$, while maintaining a double-robust property. This yields practical, distributionally robust inference for nonparametric regression and broadens applicability to modern ML estimators under covariate shift.
Abstract
This study proposes a debiasing method for smooth nonparametric estimators. While machine learning techniques such as random forests and neural networks have demonstrated strong predictive performance, their theoretical properties remain relatively underexplored. In particular, many modern algorithms lack guarantees of pointwise and uniform risk convergence, as well as asymptotic normality. These properties are essential for statistical inference and robust estimation and have been well-established for classical methods such as Nadaraya-Watson regression. To ensure these properties for various nonparametric regression estimators, we introduce a model-free debiasing method. By incorporating a correction term that estimates the conditional expected residual of the original estimator, or equivalently, its estimation error, into the initial nonparametric regression estimator, we obtain a debiased estimator that satisfies pointwise and uniform risk convergence, along with asymptotic normality, under mild smoothness conditions. These properties facilitate statistical inference and enhance robustness to covariate shift, making the method broadly applicable to a wide range of nonparametric regression problems.