A duality framework for analyzing random feature and two-layer neural networks
Hongrui Chen, Jihao Long, Lei Wu
TL;DR
The paper introduces an information-based complexity–driven duality between approximation and estimation to analyze learning with random-feature models and Barron spaces. It defines an I-complexity measure that tightly characterizes learning in noiseless settings and provides comparable lower bounds in noisy settings, while also enabling upper bounds through duality. By instantiating the framework for the spaces $F_{p,\pi}$ and Barron, the authors derive sharp, spectrum-sensitive results: (i) efficient learning of $F_{p,\pi}$ for $p>1$ beyond kernel regimes, (ii) near-optimal $L^{\infty}$ learning of RKHS via kernel ridge regression, and (iii) a unifying view bridging kernel and neural-network–oriented function classes. The methodology yields concrete rates for random-feature approximations and information-driven lower bounds that scale with kernel eigenvalue decays, offering a promising tool for broader learning analyses across settings.
Abstract
We consider the problem of learning functions within the $\mathcal{F}_{p,π}$ and Barron spaces, which play crucial roles in understanding random feature models (RFMs), two-layer neural networks, as well as kernel methods. Leveraging tools from information-based complexity (IBC), we establish a dual equivalence between approximation and estimation, and then apply it to study the learning of the preceding function spaces. The duality allows us to focus on the more tractable problem between approximation and estimation. To showcase the efficacy of our duality framework, we delve into two important but under-explored problems: 1) Random feature learning beyond kernel regime: We derive sharp bounds for learning $\mathcal{F}_{p,π}$ using RFMs. Notably, the learning is efficient without the curse of dimensionality for $p>1$. This underscores the extended applicability of RFMs beyond the traditional kernel regime, since $\mathcal{F}_{p,π}$ with $p<2$ is strictly larger than the corresponding reproducing kernel Hilbert space (RKHS) where $p=2$. 2) The $L^\infty$ learning of RKHS: We establish sharp, spectrum-dependent characterizations for the convergence of $L^\infty$ learning error in both noiseless and noisy settings. Surprisingly, we show that popular kernel ridge regression can achieve near-optimal performance in $L^\infty$ learning, despite it primarily minimizing square loss. To establish the aforementioned duality, we introduce a type of IBC, termed $I$-complexity, to measure the size of a function class. Notably, $I$-complexity offers a tight characterization of learning in noiseless settings, yields lower bounds comparable to Le Cam's in noisy settings, and is versatile in deriving upper bounds. We believe that our duality framework holds potential for broad application in learning analysis across more scenarios.