Wasserstein Distributionally Robust Nonparametric Regression
Changyu Liu, Yuling Jiao, Junhui Wang, Jian Huang
TL;DR
This paper develops a nonparametric regression framework that leverages Wasserstein distributionally robust optimization to address model misspecification and distributional shifts. By analyzing the local worst-case risk $\mathcal{R}_{\mathbb{P},k}(f;\delta)$, it shows that the Wasserstein order $k$ induces fundamentally different regularization effects: $k=1$ yields Lipschitz-type regularization while $k>1$ enforces gradient-norm regularization. The authors construct a norm-constrained neural-network estimator $\widehat{f}_{k,\delta}^{(n)}$ within a Hölder-smooth function class and prove non-asymptotic error bounds that attain near-minimax rates $n^{-2\beta/(d+2\beta)}$ (up to logs) under suitable tuning of $\delta$, with explicit trade-offs between robustness and statistical efficiency. They establish that excess local worst-case risk controls excess natural risk under distributions within the Wasserstein ambiguity set, ensuring robustness across regression and classification tasks. Numerical experiments on simulations and MNIST demonstrate improved robustness against distributional shifts and label perturbations, validating the practical impact of WDRO-NPR for high-dimensional nonparametric learning.
Abstract
Wasserstein distributionally robust optimization (WDRO) strengthens statistical learning under model uncertainty by minimizing the local worst-case risk within a prescribed ambiguity set. Although WDRO has been extensively studied in parametric settings, its theoretical properties in nonparametric frameworks remain underexplored. This paper investigates WDRO for nonparametric regression. We first establish a structural distinction based on the order $k$ of the Wasserstein distance, showing that $k=1$ induces Lipschitz-type regularization, whereas $k > 1$ corresponds to gradient-norm regularization. To address model misspecification, we analyze the excess local worst-case risk, deriving non-asymptotic error bounds for estimators constructed using norm-constrained feedforward neural networks. This analysis is supported by new covering number and approximation bounds that simultaneously control both the function and its gradient. The proposed estimator achieves a convergence rate of $n^{-2β/(d+2β)}$ up to logarithmic factors, where $β$ depends on the target's smoothness and network parameters. This rate is shown to be minimax optimal under conditions commonly satisfied in high-dimensional settings. Moreover, these bounds on the excess local worst-case risk imply guarantees on the excess natural risk, ensuring robustness against any distribution within the ambiguity set. We show the framework's generality across regression and classification problems. Simulation studies and an application to the MNIST dataset further illustrate the estimator's robustness.