MambaTron: Efficient Cross-Modal Point Cloud Enhancement using Aggregate Selective State Space Modeling
Sai Tarun Inaganti, Gennady Petrenko
TL;DR
The paper tackles view-guided point-cloud completion by enabling efficient cross-modal fusion between images and partial point clouds. It introduces MambaTron, a two-layer cell that combines a Mamba State Space Model for long-range context with a Block-Transformer for local attention, augmented by Adjacency-Preserving Reordering to maintain geometric continuity. The authors present an end-to-end, modular network with a two-stage training strategy and a loss suite including Chamfer Distance, style, projection, and 2D losses, achieving competitive results with a fraction of the parameters of prior SOTA methods on ShapeNet-ViPC and related datasets. The approach demonstrates near-linear scalability in cross-modal processing while preserving accuracy, enabling faster inference and potential applicability to downstream robotics tasks such as scene understanding and path planning.
Abstract
Point cloud enhancement is the process of generating a high-quality point cloud from an incomplete input. This is done by filling in the missing details from a reference like the ground truth via regression, for example. In addition to unimodal image and point cloud reconstruction, we focus on the task of view-guided point cloud completion, where we gather the missing information from an image, which represents a view of the point cloud and use it to generate the output point cloud. With the recent research efforts surrounding state-space models, originally in natural language processing and now in 2D and 3D vision, Mamba has shown promising results as an efficient alternative to the self-attention mechanism. However, there is limited research towards employing Mamba for cross-attention between the image and the input point cloud, which is crucial in multi-modal problems. In this paper, we introduce MambaTron, a Mamba-Transformer cell that serves as a building block for our network which is capable of unimodal and cross-modal reconstruction which includes view-guided point cloud completion.We explore the benefits of Mamba's long-sequence efficiency coupled with the Transformer's excellent analytical capabilities through MambaTron. This approach is one of the first attempts to implement a Mamba-based analogue of cross-attention, especially in computer vision. Our model demonstrates a degree of performance comparable to the current state-of-the-art techniques while using a fraction of the computation resources.