Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmentation Invariant Manifold Learning

Shulei Wang

TL;DR

This work introduces a nonlinear product-manifold model for data augmentation, positing that observed data reside on a manifold $\\mathcal{M}=T(\\mathcal{N}_s\\times\\mathcal{N}_v)$ where augmentation acts on the $\\mathcal{N}_v$ component. The authors propose augmentation-invariant manifold learning (AIML), combining augmented data through an integrated kernel to produce representations that are invariant to augmentation and reflect the intrinsic geometry of $\\mathcal{N}_s$, with two practical formulations: an augmentation-invariant Laplacian eigenmaps style and a diffusion-map style. They establish convergence of the empirical operators to Laplace-Beltrami operators on $\\mathcal{N}_s$ and derive sharp rates for downstream k-NN classification, showing that more complex augmentation (larger $d_v$) reduces effective dimensionality $d_s$ and improves classification. To scale, they present a computationally efficient stochastic-optimization formulation that learns $\\Theta_\\beta$ with unsupervised, self-supervised, and regularization signals, enabling constant-time generalization to new data. Numerical experiments on simulated manifolds and real datasets (MNIST, STL-10, ImageNet) demonstrate competitive performance against leading self-supervised methods, while providing a solid theoretical foundation for how augmentation and low-dimensional structure can jointly improve downstream analysis. The work highlights the practical and theoretical benefits of integrating data augmentation with manifold structure to obtain augmentation-invariant, geometry-aware representations.

Abstract

Data augmentation is a widely used technique and an essential ingredient in the recent advance in self-supervised representation learning. By preserving the similarity between augmented data, the resulting data representation can improve various downstream analyses and achieve state-of-the-art performance in many applications. Despite the empirical effectiveness, most existing methods lack theoretical understanding under a general nonlinear setting. To fill this gap, we develop a statistical framework on a low-dimension product manifold to model the data augmentation transformation. Under this framework, we introduce a new representation learning method called augmentation invariant manifold learning and design a computationally efficient algorithm by reformulating it as a stochastic optimization problem. Compared with existing self-supervised methods, the new method simultaneously exploits the manifold's geometric structure and invariant property of augmented data and has an explicit theoretical guarantee. Our theoretical investigation characterizes the role of data augmentation in the proposed method and reveals why and how the data representation learned from augmented data can improve the $k$-nearest neighbor classifier in the downstream analysis, showing that a more complex data augmentation leads to more improvement in downstream analysis. Finally, numerical experiments on simulated and real data sets are presented to demonstrate the merit of the proposed method.

Augmentation Invariant Manifold Learning

TL;DR

This work introduces a nonlinear product-manifold model for data augmentation, positing that observed data reside on a manifold where augmentation acts on the component. The authors propose augmentation-invariant manifold learning (AIML), combining augmented data through an integrated kernel to produce representations that are invariant to augmentation and reflect the intrinsic geometry of , with two practical formulations: an augmentation-invariant Laplacian eigenmaps style and a diffusion-map style. They establish convergence of the empirical operators to Laplace-Beltrami operators on and derive sharp rates for downstream k-NN classification, showing that more complex augmentation (larger ) reduces effective dimensionality and improves classification. To scale, they present a computationally efficient stochastic-optimization formulation that learns with unsupervised, self-supervised, and regularization signals, enabling constant-time generalization to new data. Numerical experiments on simulated manifolds and real datasets (MNIST, STL-10, ImageNet) demonstrate competitive performance against leading self-supervised methods, while providing a solid theoretical foundation for how augmentation and low-dimensional structure can jointly improve downstream analysis. The work highlights the practical and theoretical benefits of integrating data augmentation with manifold structure to obtain augmentation-invariant, geometry-aware representations.

Abstract

Data augmentation is a widely used technique and an essential ingredient in the recent advance in self-supervised representation learning. By preserving the similarity between augmented data, the resulting data representation can improve various downstream analyses and achieve state-of-the-art performance in many applications. Despite the empirical effectiveness, most existing methods lack theoretical understanding under a general nonlinear setting. To fill this gap, we develop a statistical framework on a low-dimension product manifold to model the data augmentation transformation. Under this framework, we introduce a new representation learning method called augmentation invariant manifold learning and design a computationally efficient algorithm by reformulating it as a stochastic optimization problem. Compared with existing self-supervised methods, the new method simultaneously exploits the manifold's geometric structure and invariant property of augmented data and has an explicit theoretical guarantee. Our theoretical investigation characterizes the role of data augmentation in the proposed method and reveals why and how the data representation learned from augmented data can improve the -nearest neighbor classifier in the downstream analysis, showing that a more complex data augmentation leads to more improvement in downstream analysis. Finally, numerical experiments on simulated and real data sets are presented to demonstrate the merit of the proposed method.
Paper Structure (40 sections, 15 theorems, 204 equations, 11 figures, 5 tables, 3 algorithms)

This paper contains 40 sections, 15 theorems, 204 equations, 11 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Data Augmentation in Representation Learning
  3. A Manifold Model for Data Augmentation
  4. A Peek at Augmentation Invariant Manifold Learning
  5. Contributions and Comparisons with Existing Works
  6. Augmentation Invariant Manifold Learning
  7. Classical Manifold Learning
  8. Augmentation Invariant Manifold Learning
  9. Convergence Analysis
  10. Benefits for Downstream Analysis
  11. Infinite Samples
  12. Finite Samples
  13. A Computationally Efficient Formulation
  14. Numerical Experiments
  15. Performance on Simulated Data
  16. ...and 25 more sections

Key Result

Theorem 1

Suppose that $f_v(\psi|\phi)$ is uniform distribution on $\mathcal{M}(\phi)$ for any $\phi$, there exists a constant $\kappa$ such that $1/\kappa<f_s(\phi)<\kappa$, and $f_s(\phi)$ is twice differentiable. If we choose $t=m^{-1/(d+4)}$, then where $\mathcal{L}_{\mathcal{N}_s,f_s}$ is weighted Laplace Beltrami operator $\mathcal{L}_{\mathcal{N}_s,f_s}g(\phi)=f_s^{-1}(\phi){\rm div}(f_s(\phi)\nabla

Figures (11)

  • Figure 1: An illustrative example of the product manifold: the circle with a major radius characterizes an augmentation-invariant structure, while the circle with a minor radius represents an irrelevant structure due to data augmentation (rotation). The rotation allows us to expand an observed point to a circle with a minor radius.
  • Figure 2: Plots of new data representation colored by the value of $\phi$ (left) and $\psi$ (right).
  • Figure 3: Neighborhood defined by $X$ (left) and new data representation $\Theta(X)$ (right).
  • Figure 4: The embedding of three different product manifolds in $\mathbb{R}^2$. Different columns corresponds to different augmentation invariant manifold learning methods. All figures are colored by $\phi$.
  • Figure 5: Comparisons of two formulations of augmentation invariant manifold learning. The left figure summarizes the misclassification rate of different methods, and the right figure reports the computation time of different methods. The estimated slopes in the right figure are: 0.87 (Algorithm 2, 25 epochs), 0.92 (Algorithm 2, 50 epochs), 0.96 (Algorithm 2, 100 epochs), 1.97 (Algorithm 1, $n=3$), 1.99 (Algorithm 1, $n=5$).
  • ...and 6 more figures

Theorems & Definitions (25)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem S1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 15 more