Spectral Informed Mamba for Robust Point Cloud Processing

TL;DR

This work extends state-space models to 3D point clouds by introducing a spectral-informed Mamba framework. It capitalizes on the Laplacian spectrum of a patch-connectivity graph to achieve isometry-invariant token ordering (SAST), a recursive spectral partitioning scheme for accurate point-level segmentation (HLT), and a tour-preserving token placement strategy for Masked Autoencoders (TAR). Empirical results show consistent gains in object classification, segmentation, and few-shot scenarios while maintaining favorable computational efficiency. Overall, the approach offers a robust, geometry-aware alternative to grid-based traversals for point-cloud processing with Mamba networks.

Abstract

State space models have shown significant promise in Natural Language Processing (NLP) and, more recently, computer vision. This paper introduces a new methodology leveraging Mamba and Masked Autoencoder networks for point cloud data in both supervised and self-supervised learning. We propose three key contributions to enhance Mamba's capability in processing complex point cloud structures. First, we exploit the spectrum of a graph Laplacian to capture patch connectivity, defining an isometry-invariant traversal order that is robust to viewpoints and better captures shape manifolds than traditional 3D grid-based traversals. Second, we adapt segmentation via a recursive patch partitioning strategy informed by Laplacian spectral components, allowing finer integration and segment analysis. Third, we address token placement in Masked Autoencoder for Mamba by restoring tokens to their original positions, which preserves essential order and improves learning. Extensive experiments demonstrate the improvements of our approach in classification, segmentation, and few-shot tasks over state-of-the-art baselines.

Figures (11)

• Figure 1: (a) *sast applied to the patched point clouds of a mesh surface. (b) Traversal from left to right, based on the first to fourth non-constant smallest eigenvectors. (c) Traversal based on the largest eigenvector, forming a non-continuous sequence of tokens.
• Figure 2: Overview of the proposed sst method. (a) Point cloud, (b) Patchification, (c) Forming the adjacency graph, (d) Traversal based on sast using $s$ non-constant smallest eigenvectors, (e) hlt for segmentation tasks, (f) tar strategy for Masked Autoencoders. The process includes reverse and concatenation operations, with learnable tokens, representations, and masked tokens highlighted. (g) The classification task involves sorting tokens by different eigenvectors, concatenating them, and then feeding them into the network. (h) The segmentation task where HLT is applied on the tokens ($\vec{q}$) and $\vec{q}$ is fed into the network. (i) A flowchart visualizing the techniques used in self-supervised learning and various downstream tasks.
• Figure 3: (a) Visualization of the four non-constant smallest Laplacian eigenvectors ($v^{(k)}$, $k=1,\ldots,4$) and (b) the discrete partitioning ($q$) of our HLT strategy combining the information of all four eigenvectors. Note: we assumed that patches contain a single point for better visualization.
• Figure 4: Analysis of the number of non-constant smallest eigenvectors and comparison with previous methods (Left) and analysis of the number of nearest neighbors $K$ (Right).
• Figure 5: The effect of the tar strategy in the pretraining phase (Left) and in finetuning (Right).