PillarMamba: Learning Local-Global Context for Roadside Point Cloud via Hybrid State Space Model

TL;DR

PillarMamba tackles roadside point-cloud 3D object detection, where dense context and long-range spatial relations are critical. It introduces Cross-stage State-space Group (CSG) for efficient global context and Hybrid State-space Block (HSB) to preserve local structure and historical memory, augmented by a 2D-Selective-Scan mechanism that employs discrete state-space modeling over BEV sequences. The architecture uses a Pillar Feature Encoder to create dense BEV maps, a dense backbone with multi-scale fusion, and a CenterPoint-style detection head, achieving state-of-the-art results on the DAIR-V2X-I roadside benchmark. This work advances ITS/V2X perception by enabling robust roadside detection with favorable accuracy-latency trade-offs, and its modular design facilitates future improvements and deployment on edge hardware.

Abstract

Serving the Intelligent Transport System (ITS) and Vehicle-to-Everything (V2X) tasks, roadside perception has received increasing attention in recent years, as it can extend the perception range of connected vehicles and improve traffic safety. However, roadside point cloud oriented 3D object detection has not been effectively explored. To some extent, the key to the performance of a point cloud detector lies in the receptive field of the network and the ability to effectively utilize the scene context. The recent emergence of Mamba, based on State Space Model (SSM), has shaken up the traditional convolution and transformers that have long been the foundational building blocks, due to its efficient global receptive field. In this work, we introduce Mamba to pillar-based roadside point cloud perception and propose a framework based on Cross-stage State-space Group (CSG), called PillarMamba. It enhances the expressiveness of the network and achieves efficient computation through cross-stage feature fusion. However, due to the limitations of scan directions, state space model faces local connection disrupted and historical relationship forgotten. To address this, we propose the Hybrid State-space Block (HSB) to obtain the local-global context of roadside point cloud. Specifically, it enhances neighborhood connections through local convolution and preserves historical memory through residual attention. The proposed method outperforms the state-of-the-art methods on the popular large scale roadside benchmark: DAIR-V2X-I. The code will be released soon.

Figures (3)

• Figure 1: We assume a roadside point cloud scene with vehicles in BEV map, where the orange and blue grids denote the vehicle A and B. In this scene, vehicle A and B are in close neighborhood in 3D space. When we use visual Mamba to process the BEV map of the current scene, it divides the grids in the map into 4 scans, called Cross-Scan. The traditional visual Mamba methods simply flatten the tokens of BEV map into 1D sequences by scanning them row-by-row or column-by-column. It not only increases the distance between neighborhood tokens in BEV map, and also small objects to be drowned in historical invalid information of recursive equations, which weakens the local dependencies and blurs the foreground and background.
• Figure 2: The proposed PillarMamba consists of Pillar Feature Encoder (PFE), Backbone and Detection Head. We first convert the roadside point cloud into dense BEV feature map $F_0 \in R^{C\times X\times Y}$ based on PFE. We use Cross-stage State-space Group (CSG) as the basic feature extractor in dense backbone network. $F_1 \in R^{C\times X\times Y}$ is obtained by CSG with 2 HSB layers. Then, $F_1$ is fed into a down-sample layer and CSG with 2 HSB layers, obtain the $F_2 \in R^{C\times \frac{X}{2}\times \frac{Y}{2}}$, which is used to further conserve memory. As above, we obtain $F_3$ and $F_4$ in turn. Then, we concatenate $F_2$, $F_3$ and $F_4$ in channel dimension to integrate multi-scale information. We then feed the features into the up-sample layer to get the $F_5 \in R^{C\times X\times Y}$. Detection Head first encodes the pillar features with a convolutional layer and then predicts the 3D bounding box consisting of locations $(x, y, z)$, dimensions $(l, w, h)$ and yaw angle $\theta$.
• Figure 3: We visualize some scenes in BEV map to qualitatively analysis the comparison between our proposed PillarMamba and the state-of-the-art methods (e.g. PillarNet and PillarNeXt). The color distribution of the point cloud is related to the x-axis coordinates, with blue bounding boxes that denote the ground truth and red bounding boxes that denote the predictions of network. It can be seen that, in both scene A and B, PillarNet and PillarNeXt produce incorrect predictions due to the noisy and sparse point cloud at long distances. However, our proposed PillarMamba makes correct predictions in the detection of long-distance in these scenes and correctly classifies sparse point cloud with ambiguous shapes.