Several Supporting Evidences for the Adaptive Feature Program

TL;DR

The paper tackles the theoretical understanding of neural network generalization by proposing the Adaptive Feature Program (AFP), which jointly learns feature representations and linear readouts through gradient flow. It introduces the Feature Error Measure (FEM) to quantify how well learned features align with the target function, and models training via overparametrized sequence models justified by Le Cam equivalence. The authors develop detailed analyses for diagonal fixed-basis and directional single-index and multi-index models, showing FEM decreases and often achieves near-optimal nonparametric rates, with clear phase dynamics and dependencies on information indices like $\rz$. They also establish path-equivalence results linking sequence-model dynamics to empirical-loss dynamics, supported by numerical studies. Overall, the work provides a unified framework that connects classical statistical understanding with modern feature-learning dynamics, offering insights into how adaptive representations can improve generalization in high-dimensional regimes.

Abstract

Theoretically exploring the advantages of neural networks might be one of the most challenging problems in the AI era. An adaptive feature program has recently been proposed to analyze the feature learning characteristic property of neural networks in a more abstract way. Motivated by the celebrated Le Cam equivalence, we advocate the over-parametrized sequence models to further simplify the analysis of the training dynamics of adaptive feature program and present several supporting evidences for the adaptive feature program. More precisely, after having introduced the feature error measure (FEM) to characterize the quality of the learned feature, we show that the FEM is decreasing during the training process of several concrete adaptive feature models including linear regression, single/multiple index models, etc. We believe that this hints at the potential successes of the adaptive feature program.

Figures (4)

• Figure 1: The program of this paper. We propose to model complex neural networks with adaptive feature program, capturing its dynamic feature learning. Moreover, we propose to analyze the adaptive features under the sequence model observation, which allows us to focus on the training dynamics while preserving the essence of non-parametric regression.
• Figure 2: Decay of feature error measure $\mathcal{E}^*$ (FEM) during the training process. Upper row: diagonal adaptive feature (Diag); lower row: directional adaptive feature for single-index model (SIM). Left column: empirical loss; right column: sequence loss. The shaded regions represent the standard deviation computed by 200 runs.
• Figure 3: Similarity between the training curves under the empirical loss $\mathcal{L}_n$ and sequence loss $\bar{\mathcal{L}}_n$. We plot the energy distances estimated from 200 independent runs, and also shaded regions represent the standard deviation estimated by bootstrapping. Upper row: $D(\hat{f}^{\text{Seq}}_t,\hat{f}^{\text{GD}}_t)$ is much smaller than that of $D(\hat{f}^{\text{Seq}}_t,0)$, $D(\hat{f}^{\text{GD}}_t,0)$ along the training path. Lower row: The difference between $\hat{f}^{\text{GD}}_t$ and $\hat{f}^{\text{Seq}}_t$ decreases as $n$ increases. The methods in three columns are fixed feature method, diagonal adaptive kernel method and directional adaptive feature method respectively.
• Figure 4: Energy distances between the feature error measure $\mathcal{E}^*$ (FEM) under the empirical loss $\mathcal{L}_n$ and sequence loss $\bar{\mathcal{L}}_n$.

Theorems & Definitions (126)

• Definition 1.1
• Remark 1.2
• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Theorem 2.4: SIM Population Dynamics
• Theorem 2.5
• Corollary 2.6
• Theorem 2.7: Population Dynamics
• Theorem 2.8