Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

TL;DR

The paper develops a dynamic, data-adaptive RKHS framework for two-layer neural networks trained by gradient flow. It proves that, at stationarity, the network's learned representation is the optimal projection of the target function onto a data-adaptive RKHS, while the residual lies in a smaller subspace than the RKHS kernel’s orthogonal complement. It introduces a GD-driven kernel $\mathcal{K}_\infty$ and demonstrates a nontrivial gap between $\mathcal{H}_\infty$ and $\mathcal{K}_\infty$, providing a representation-based explanation for interpolation and generalization in neural networks. The work also connects gradient-flow dynamics to kernel ridgeless regression with adaptive kernels and characterizes the time evolution of the underlying kernels, offering a path to adaptive generalization theory and practical insights for representation learning. Together, these results offer a principled, nonparametric view of how neural networks adapt their representations to data and tasks during training, with implications for optimization, generalization, and algorithmic efficiency.

Abstract

Consider the problem: given the data pair $(\mathbf{x}, \mathbf{y})$ drawn from a population with $f_*(x) = \mathbf{E}[\mathbf{y} | \mathbf{x} = x]$, specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does $f_t$, the function computed by the neural network at time $t$, relate to $f_*$, in terms of approximation and representation? What are the provable benefits of the adaptive representation by neural networks compared to the pre-specified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks. Firstly, we show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation and performs the global least-squares projection onto the adaptive RKHS, simultaneously. Secondly, we prove that as the RKHS is data-adaptive and task-specific, the residual for $f_*$ lies in a subspace that is potentially much smaller than the orthogonal complement of the RKHS. The result formalizes the representation and approximation benefits of neural networks. Lastly, we show that the neural network function computed by gradient flow converges to the kernel ridgeless regression with an adaptive kernel, in the limit of vanishing regularization. The adaptive kernel viewpoint provides new angles of studying the approximation, representation, generalization, and optimization advantages of neural networks.

Figures (5)

• Figure 1: Illustration of Theorem \ref{['thm:proj-solution']}. Red dotted line denotes the function $f_t$ computed along the gradient flow dynamics on the weights of NN. Along training, one learns a sequence of dynamic RKHS representation $\mathcal{H}_t$'s. Over time, $f_t$ converges to the projection of $f_*$ onto $\mathcal{H}_\infty$. We emphasize that the initial function $f_0$ computed by NN is very different from the projection of $f_*$ onto the initial RKHS $\mathcal{H}_0$.
• Figure 2: Illustration of \ref{['thm:gap-decomposition']}: fixed basis vs. adaptive learned basis. In classic statistics, one specifies the fixed function space/basis $H_0$ then decompose $f_*$ into the projection $\hat{f}_0$ and residual $\Delta_0 \in \text{Ker}(H_0)$. However, for GD on NN, one learns the adaptive basis $H_\infty$ that depends on $f_*$. Therefore, the residual $\Delta_\infty$ lies in a subspace of $\text{Ker}(H_\infty)$.
• Figure 3: Log of the sorted top $80\%$ eigenvalues of kernel matrix along training with different $f_*$
• Figure 4: Log of the sorted top $80\%$ eigenvalues of kernel matrix along training with random labels.
• Figure 5: Log of sorted top $90\%$ eigenvalues of kernel matrix along training process for mnist

Theorems & Definitions (30)

• Theorem 3.1: Approximation
• Remark 3.1
• Corollary 3.1: ERM
• Theorem 3.2: Representation Benefits
• Remark 3.2
• Proposition 4.1: Interpolation: Connection to Kernel Ridgeless Regression
• Proposition 4.2: Adaptive Generalization
• Lemma 5.1: Dynamic kernel of finite neurons GD
• Corollary 5.1
• Proposition 5.1: Balanced condition