Conditional regression for the Nonlinear Single-Variable Model
Yantao Wu, Mauro Maggioni
TL;DR
The paper introduces the Nonlinear Single-Variable Model (NSVM) for high-dimensional regression, where the regression function is F(X)=f(Π_γ X) with an unknown curved projection γ and a one-dimensional link f. It develops a nonparametric estimator based on conditional (inverse) regression that recovers the geometry of γ, the nonlinear projection Π_γ, and the outer function f, all while avoiding the curse of dimensionality; the estimator runs in near-linear time in the sample size and achieves a near-minimax rate for one-dimensional regression, up to log factors, with a controllable curve-approximation error. The theoretical analysis provides concentration bounds for slice-based parameter estimates, distance-based slice assignment accuracy, and a detailed MSE decomposition, culminating in guarantees that the overall error matches the 1D rate under mild, dimension-free assumptions on f and γ. Numerical experiments on circular arcs and Meyer helix curves, plus an application to reaction-path committor learning in Langevin dynamics, demonstrate robustness to curvature and dimensionality, and illustrate the estimator’s interpretability via local tangent estimates to γ. The work suggests that, by exploiting nonlinear compositional structure with a one-dimensional manifold, one can effectively learn high-dimensional functions without incurring exponential dependence on ambient dimension, with potential extensions to higher-dimensional manifolds and non-monotone link functions.
Abstract
Regressing a function $F$ on $\mathbb{R}^d$ without the statistical and computational curse of dimensionality requires special statistical models, for example that impose geometric assumptions on the distribution of the data (e.g., that its support is low-dimensional), or strong smoothness assumptions on $F$, or a special structure $F$. Among the latter, compositional models $F=f\circ g$ with $g$ mapping to $\mathbb{R}^r$ with $r\ll d$ include classical single- and multi-index models, as well as neural networks. While the case where $g$ is linear is well-understood, less is known when $g$ is nonlinear, and in particular for which $g$'s the curse of dimensionality in estimating $F$, or both $f$ and $g$, may be circumvented. Here we consider a model $F(X):=f(Π_γX)$ where $Π_γ:\mathbb{R}^d\to[0,\textrm{len}_γ]$ is the closest-point projection onto the parameter of a regular curve $γ:[0, \textrm{len}_γ]\to\mathbb{R}^d$, and $f:[0,\textrm{len}_γ]\to \mathbb{R}^1$. The input data $X$ is not low-dimensional: it can be as far from $γ$ as the condition that $Π_γ(X)$ is well-defined allows. The distribution $X$, the curve $γ$ and the function $f$ are all unknown. This model is a natural nonlinear generalization of the single-index model, corresponding to $γ$ being a line. We propose a nonparametric estimator, based on conditional regression, that under suitable assumptions, the strongest of which being that $f$ is coarsely monotone, achieves, up to log factors, the $\textit{one-dimensional}$ optimal min-max rate for non-parametric regression, up to the level of noise in the observations, and be constructed in time $\mathcal{O}(d^2 n\log n)$. All the constants in the learning bounds, in the minimal number of samples required for our bounds to hold, and in the computational complexity are at most low-order polynomials in $d$.