An Incremental Non-Linear Manifold Approximation Method

TL;DR

This work introduces Incremental GMRA, a streaming non-linear dimension reduction method based on Geometric Multi-resolution Analysis that incrementally updates a multiscale manifold approximation as new data arrive. By leveraging a Cover Tree for data partitioning and Brand-type SVD/covariance updates, the method maintains the GMRA structure and updates PCA bases and wavelet coefficients in real time, with MSE-driven cluster splitting to control model complexity. Theoretical error analyses bound the impact of incremental updates on singular values and subspace angles, while numerical experiments on Swiss Roll and intersecting-manifold scenarios demonstrate accurate, adaptive manifold representation with scalable computation. The approach enables real-time visualization and interactive graphics where high-dimensional data evolve over time, offering robust incremental learning with multiscale, non-linear structure preservation.

Abstract

Analyzing high-dimensional data presents challenges due to the "curse of dimensionality'', making computations intensive. Dimension reduction techniques, categorized as linear or non-linear, simplify such data. Non-linear methods are particularly essential for efficiently visualizing and processing complex data structures in interactive and graphical applications. This research develops an incremental non-linear dimension reduction method using the Geometric Multi-Resolution Analysis (GMRA) framework for streaming data. The proposed method enables real-time data analysis and visualization by incrementally updating the cluster map, PCA basis vectors, and wavelet coefficients. Numerical experiments show that the incremental GMRA accurately represents non-linear manifolds even with small initial samples and aligns closely with batch GMRA, demonstrating efficient updates and maintaining the multiscale structure. The findings highlight the potential of Incremental GMRA for real-time visualization and interactive graphics applications that require adaptive high-dimensional data representations.

Figures (5)

• Figure 1: An example illustration of a tree decomposition of point cloud data and hyperplane approximations at each tree node. At the root level, all the sample points are contained within a single node. At the second level, the samples split into two nodes, and the leaf level comprises five tree nodes. Each tree node is represented in a different color.
• Figure 2: Figure (a) shows the low-dimensional GMRA approximation for the 500 training data. Figure (b) displays the incrementally updated low-dimensional approximation following the inclusion of the 2000 more samples, and figure (c) shows the final GMRA approximation after streaming in all the remaining samples. Figure (d) -- Low dimensional approximation when all 50,000 sample points are used to create GMRA structure (Ground truth)
• Figure 3: Figure (a) shows the average mean squared error of the approximation (blue) by sample size for the Swiss Roll data, along with the average maximum MSE (red) at leaf clusters after 30 repeats. The standard error bars are also depicted on each curve. A green horizontal line depicts the threshold level $\epsilon = 0.1$. Figure (b) shows the boxplots of the number of leaf clusters used for the approximation by sample size after 30 repeats. The line bisecting the box represents the median and the $+$ sign within the box represents the mean number of leaf clusters. Figure (c) illustrates the maximum height of the cluster map. Each boxplot in Figure (c) illustrates, on average, how many new samples it takes before the cluster map needs to resolve an additional depth (based on 30 repeats of the experiment). Figure (d) depicts the boxplots of the number of samples within the leaf cluster where the maximum MSE is observed after each increment (the orange horizontal line shows $M = 30$).
• Figure 4: Figure (a) shows the Swiss roll with a linear hyperplane intersected. Figure (b) shows the approximation for the 1000 training samples from the Swiss roll manifold. Figure (c) shows the incrementally updated low-dimensional approximation following the inclusion of all 60,000 samples (50,000 from Swiss roll and 10,000 from plane). Figure (d) --GMRA construction for the entire data set of linear hyperplane intersecting a Swiss roll (Ground truth)
• Figure 5: Figure (a) shows the average mean squared error of the approximation (blue) by sample size for the experiment of Swiss Roll intersected with a plane, along with the average maximum MSE (red) at leaf clusters after 30 repeats. The standard error bars are also depicted on each curve. A green horizontal line depicts the threshold level $\epsilon = 0.1$. Figure (b) shows the boxplots of the number of leaf clusters used for the approximation by sample size after 30 repeats. The line bisecting the box represents the median, and the $+$ sign within the box represents the mean number of leaf clusters. Figure (c) illustrates the maximum height of the cluster map. Each boxplot in Figure (c) illustrates, on average, how many new samples it takes before the cluster map needs to resolve an additional depth (based on 30 repeats of the experiment). Figure (d) depicts the boxplots of the number of samples within the leaf cluster where the maximum MSE is observed after each increment (the orange horizontal line shows $M = 30$).

Theorems & Definitions (9)

• Remark 3.1
• Remark 3.2
• Remark 3.3
• Remark 3.4
• Lemma 1
• Proposition 1
• Lemma 2
• proof
• proof