Local EGOP for Continuous Index Learning

TL;DR

It is proved that Local EGOP learning adapts to the regularity of the function of interest, showing that under a supervised noisy manifold hypothesis, intrinsic dimensional learning rates are achieved for arbitrarily high-dimensional noise.

Abstract

We introduce the setting of continuous index learning, in which a function of many variables varies only along a small number of directions at each point. For efficient estimation, it is beneficial for a learning algorithm to adapt, near each point $x$, to the subspace that captures the local variability of the function $f$. We pose this task as kernel adaptation along a manifold with noise, and introduce Local EGOP learning, a recursive algorithm that utilizes the Expected Gradient Outer Product (EGOP) quadratic form as both a metric and inverse-covariance of our target distribution. We prove that Local EGOP learning adapts to the regularity of the function of interest, showing that under a supervised noisy manifold hypothesis, intrinsic dimensional learning rates are achieved for arbitrarily high-dimensional noise. Empirically, we compare our algorithm to the feature learning capabilities of deep learning. Additionally, we demonstrate improved regression quality compared to two-layer neural networks in the continuous single-index setting.

Figures (7)

• Figure 1: Localizations from Local EGOP Learning (Algorithm 1) centered at each of the highlighted points (in red) trained on a grayscale image of a mandrill bc9m-f507-21. Here $X$ is the pixel location and $Y$ the grayscale value of the image. For visualization purposes the procedure was stopped early to enforce large localizations, and the 125 highest weight pixels are highlighted at each point. On the right the image is magnified to the highlighted region boxed-off on the left.
• Figure 2: Supervised noisy manifold data in $\mathbb{R}^2$. In a neighborhood about a closed curve we overlay a heatmap of a function invariant to normal displacement. The curve is displayed in black, the normal spaces are highlighted in white, and red gradients are displayed at randomly sampled points, with lengths proportionate to the gradient magnitude.
• Figure 3: Local EGOP Learning and a deep transformer architecture applied to data sampled from an annulus. Labels are generated with no dependence on the radius in the parameterization, with these values overlaid as a heatmap. About the point $x^* = (1,0)$, points are displayed with opacity $w \propto \exp(-d(x^*,x_i)^2)$, where $d(\cdot, \cdot)$ is the AGOP Mahalanobis distance (top) and transformer feature embedding (bottom). Both regions are displayed after progressively many iterations/training batches.
• Figure 4: Comparison of the eigenvalue decay of $\Sigma_i$ for Local EGOP Learning with and without momentum. On the left, we set $\beta=0.7$, on the right $\beta = 1$ (no momentum). Without momentum, the second order eigenvalues are prone to wild oscillations.
• Figure 5: Eigenvalue decay of $\Sigma_i$ with Local EGOP Learning applied to data satisfying the noisy manifold hypothesis about $S^2$. The orthogonal exhibits light decay, approaching second order anisotropy asymptotically.

Theorems & Definitions (80)

• Lemma 1: Poincaré Inequality
• Lemma 2: Variance
• Theorem 1: Local MSE
• Proposition 1: Alternating Minimization
• Lemma 3: Generic Taylor Expansion
• Example 1: Scalar Recurrence
• Example 2: Momentum
• Theorem 2: Second Order Anisotropy
• Lemma 4: Noisy Manifold Taylor Expansion
• Lemma 5: Gradient Geometry