Stable Filtering for Efficient Dimensionality Reduction of Streaming Manifold Data

TL;DR

This work introduces Randomized Filtering (RF), a training-free, online dimensionality reduction technique designed to preserve nonlinear manifold geometry in streaming, high-dimensional data. RF uses random sign flipping, the fast Fourier transform, and random Fourier coefficient subsampling to form a stable embedding with guarantees akin to the restricted isometry property, requiring a number of measurements that scales linearly with the manifold dimension and logarithmically with ambient dimension. Empirical results across simulated and real datasets in neuroscience and fluid dynamics show RF better preserves data geometry than low-pass filtering and PCA, enabling accurate event detection, manifold learning, and phase classification at high compression. The approach offers practical benefits for real-time data processing and potential hardware implementations, with broad applicability to regression, classification, and manifold-learning tasks in data-rich scientific domains.

Abstract

Many areas in science and engineering now have access to technologies that enable the rapid collection of overwhelming data volumes. While these datasets are vital for understanding phenomena from physical to biological and social systems, the sheer magnitude of the data makes even simple storage, transmission, and basic processing highly challenging. To enable efficient and accurate execution of these data processing tasks, we require new dimensionality reduction tools that 1) do not need expensive, time-consuming training, and 2) preserve the underlying geometry of the data that has the information required to understand the measured system. Specifically, the geometry to be preserved is that induced by the fact that in many applications, streaming high-dimensional data evolves on a low-dimensional attractor manifold. Importantly, we may not know the exact structure of this manifold a priori. To solve these challenges, we present randomized filtering (RF), which leverages a specific instantiation of randomized dimensionality reduction to provably preserve non-linear manifold structure in the embedded space while remaining data-independent and computationally efficient. In this work we build on the rich theoretical promise of randomized dimensionality reduction to develop RF as a real, practical approach. We introduce novel methods, analysis, and experimental verification to illuminate the practicality of RF in diverse scientific applications, including several simulated and real-data examples that showcase the tangible benefits of RF.

Figures (6)

• Figure 1: Illustration of randomized filtering (RF) which consists of three steps: 1. randomize the signs of the input vector; 2. compute the FFT of the result; 3. randomly subsample the Fourier coefficients. RF maps points from a $D$ dimensional manifold residing in $\mathbb{R}^N$ to the reduced space $\mathbb{R}^M$ where $M<N$. Our theory guarantees that for sufficiently large $M$, the mapping is a stable embedding, i.e., pairwise distances are approximately preserved. Since the main computational step involves the FFT, the algorithm is of complexity $\mathcal{O}\left(N \log N\right)$.
• Figure 2: The isometry constant $\delta$ quantifies how well an embedding preserves the geometry of the input space. A value of $\delta = 0$ corresponds to a perfect embedding where distances between all pairs of input points are equal to corresponding distances in the reduced space. Shown are estimates of the isometry constant for several synthetic datasets (cardiac model Elshrif2014QuantitativeComparisonBehavior, the neural imaging data used in Fig. \ref{['fig:ca_sim']}, the sine manifold Eftekhari2015NewAnalysisManifold, and solutions to the vorticity equations as in Fig. \ref{['fig:vorticity']}) as well as real datasets (voltage sensitive dye imaging from rodent experiments Zheng2015AdaptiveShapingCortical and functional magnetic resonance imaging Nooner2012NKIRocklandSampleModel). Lower isometry constants may be achieved with far fewer measurements with RF (left) compared to LPF (right) which lacks similar stability guarantees.
• Figure 3: (a) Calcium imaging uses florescent calcium indicators to measure activity in the brain. (b) Example frame of simulated calcium imaging data. (c) In LPF, each frame is blurred and downsampled by $15$X in each spatial dimension as a crude form of dimensionality reduction. The fine details of the image are lost, especially in areas containing overlapping cells. (d-f) Performance comparison for event detection in synthetic calcium imaging data. Events are estimated by thresholding the output of Equation \ref{['eqn:ip']}, and the $F_1$-score is used as the performance metric (higher is better). (d) RF produces favorable results even after heavy compression. (e) RF outperforms low-pass filtering (LPF) up to moderate levels of noise. (f) As the number of overlapping cells increases, LPF cannot distinguish activity between nearby or overlapping cells. In contrast, RF can separate activity between cells with significant levels of overlap.
• Figure 4: (a) Mouse neural fluorescence data, 34600 time points recorded at 9.6 Hz from 6519 neurons Manley2024VaziriMouseData. (b) Example neurons selected from the data. (c) RF preserves pairwise distances better than PCA for mouse brain data across a range of compression ratios, as measured by the isometry constant $\delta$. PCA results are averaged across 100 samples (without replacement, 80 percent of the data, error bars: standard deviation). Randomized filtering results are averaged across 10 samples, each run through 10 random seeds of randomized filtering. (d) RF compression better preserves overall embedded manifold shapes of the data than PCA compression. We embedded each of the 34600 time points from the original data (6519 dimensions), and the RF and PCA compressed data into 4 embedding dimensions using LLE separately for each data matrix. We compared the manifold shapes in the 4-D space using the Procrustes distance between the data embedding and 4-D embeddings of RF- and PCA-compressed data (compression ratios on the x-axis). A Procrustes distance of 0 indicates no difference in shape between the two embeddings. RF preserves the overall 4-D manifold shape better than PCA at compressions up to approximately 100x (65 dimensions).
• Figure 5: (a) Vorticity is a quantity that describes the local tendency of a fluid to rotate, and can be measured experimentally using, e.g., particle image velocimetry as shown. (b) Example frame of the solution to the vorticity Equations \ref{['eqn:vorticity']}. (c) Example dimensionality reduction by a factor of $15$ in each dimension via LPF and downsampling. As expected, the high-frequency content has vanished. (d-f) Performance in a phase classification task on solutions to the vorticity equations. The task is to classify which phase of the forcing function was used to generate the solution based on compressed measurements of the solution. (d) RF achieves a low classification error rate even after 2 to 3 orders of magnitude of compression. (e) RF produces superior phase classification performance in noisy conditions, whereas information necessary for classification is completely lost under low-pass filtering. (f) RF allows for correct classification more often than LPF even when the problem is made more difficult by drawing phases from a larger set of candidates.

Theorems & Definitions (3)

• Theorem 2
• proof
• Remark