Improving Adaptivity via Over-Parameterization in Sequence Models

TL;DR

An over-parameterized gradient descent in the realm of sequence model is introduced to capture the effects of various orders of a fixed set of eigen-functions to explore the impact of varying eigenfunction orders.

Abstract

It is well known that eigenfunctions of a kernel play a crucial role in kernel regression. Through several examples, we demonstrate that even with the same set of eigenfunctions, the order of these functions significantly impacts regression outcomes. Simplifying the model by diagonalizing the kernel, we introduce an over-parameterized gradient descent in the realm of sequence model to capture the effects of various orders of a fixed set of eigen-functions. This method is designed to explore the impact of varying eigenfunction orders. Our theoretical results show that the over-parameterization gradient flow can adapt to the underlying structure of the signal and significantly outperform the vanilla gradient flow method. Moreover, we also demonstrate that deeper over-parameterization can further enhance the generalization capability of the model. These results not only provide a new perspective on the benefits of over-parameterization and but also offer insights into the adaptivity and generalization potential of neural networks beyond the kernel regime.

Figures (6)

• Figure 1: Comparison of the generalization error rates between vanilla gradient descent and over-parameterized gradient descent (OpGD). We set $p=1$ and $q=2$ for the truth parameter $\bm{\theta}^*$, and $\gamma=1.5$ for the left column and $\gamma=3$ for the right column. For each $n$, we repeat $64$ times and plot the mean and the standard deviation.
• Figure 2: The evolution of the trainable eigenvalues $a_j(t) b_j^D(t)$ over the time $t$ across components $j=100$ to $200$ for $D=1$. The blue line shows the eigenvalues and the black marks show the non-zero signals scaled according to prop:EigLearn. For the settings, we set $p=1$, $q=2$ and $\gamma=2$.
• Figure 3: Comparison of the generalization error rates between vanilla gradient descent and over-parameterized gradient descent (OpGD). We set $p=1$ and $q=2$ for the truth parameter $\bm{\theta}^*$. The left and right columns show respectively the generalization error and the orcale stopping time with respect to $n$. For the upper row, we set the eigenvalue decay rate $\gamma = 1.5$; for the lower row, we set $\gamma = 3$. For each $n$, we repeat $64$ times and plot the mean and the standard deviation.
• Figure 4: The generalization error as well as the evolution of the eigenvalue terms $a_j(t)b_j^D(t)$ over the time $t$. The first row shows the generalization error of three parameterizations $D=0,1,3$ with respect to the training time $t$. The rest of the rows show the evolution of the eigenvalue terms $a_j(t)b_j^D(t)$ over the time $t$. For presentation, we select the index $j=100$ to $200$. The blue line shows the eigenvalue terms and the black marks show the non-zero signals scaled according to prop:EigLearn. For the settings, we set $p=1$, $q=2$ and $\gamma=2$.
• Figure 5: Comparison of the generalization error between the fixed kernel gradient method and the diagonal adaptive kernel method. The left figure shows the generalization error curve of a single trial. The right figure shows the generalization error rates with respect to the sample size $n$.

Theorems & Definitions (19)

• Example 2.1: Eigenfunctions in common order
• Example 2.2: Low-dimensional structure
• Example 2.3: Misalignment
• Theorem 3.1
• Theorem 3.2
• Corollary 3.3
• Proposition 3.4
• Lemma D.1
• proof
• Lemma D.2: Noise case