State Space Model Meets Transformer: A New Paradigm for 3D Object Detection

TL;DR

This paper tackles the limitation of DETR-based 3D indoor object detectors caused by fixed scene point features in decoder layers. It introduces DEST, a State Space Model–driven paradigm where object queries are modeled as system states and scene points as inputs, enabling simultaneous updates with linear complexity $O(N)$. The core contributions are the Interactive State Space Model (ISSM) and an ISSM-based decoder featuring Hilbert-based serialization, bidirectional scanning, inter-state attention, and a gated FFN, all tailored for unordered point clouds. Extensive experiments on ScanNet V2 and SUN RGB-D demonstrate consistent improvements over strong baselines (GroupFree and VDETR) and establish new state-of-the-art results, with ablations validating the effectiveness of each component. The DEST framework offers a promising direction for efficient, high-performance 3D object detection and potential extension to other point-cloud tasks.

Abstract

DETR-based methods, which use multi-layer transformer decoders to refine object queries iteratively, have shown promising performance in 3D indoor object detection. However, the scene point features in the transformer decoder remain fixed, leading to minimal contributions from later decoder layers, thereby limiting performance improvement. Recently, State Space Models (SSM) have shown efficient context modeling ability with linear complexity through iterative interactions between system states and inputs. Inspired by SSMs, we propose a new 3D object DEtection paradigm with an interactive STate space model (DEST). In the interactive SSM, we design a novel state-dependent SSM parameterization method that enables system states to effectively serve as queries in 3D indoor detection tasks. In addition, we introduce four key designs tailored to the characteristics of point cloud and SSM: The serialization and bidirectional scanning strategies enable bidirectional feature interaction among scene points within the SSM. The inter-state attention mechanism models the relationships between state points, while the gated feed-forward network enhances inter-channel correlations. To the best of our knowledge, this is the first method to model queries as system states and scene points as system inputs, which can simultaneously update scene point features and query features with linear complexity. Extensive experiments on two challenging datasets demonstrate the effectiveness of our DEST-based method. Our method improves the GroupFree baseline in terms of AP50 on ScanNet V2 (+5.3) and SUN RGB-D (+3.2) datasets. Based on the VDETR baseline, Our method sets a new SOTA on the ScanNetV2 and SUN RGB-D datasets.

Figures (13)

• Figure 1: (a): Transformer decoder solely updates the features of the query points, while our ISSM-based decoder simultaneously updates the features of scene points and query points. (b): The DETR-based models show only slight accuracy enhancements in the later layers, whereas the DEST-based methods significantly boost the performance in the later layers.
• Figure 2: The overall framework of the DEST-based method for 3D object detection. We first utilize an encoder to extract 3D features, followed by a state sampling module to select state points, referred to as queries in DETR architecture. Subsequently, we input both the scene points and state points into the ISSM-based decoder for simultaneous updates. Finally, the updated state points are fed into a detection head to predict the 3D bounding boxes.
• Figure 3: Overview of the Interactive State Space Model. In the ISSM, we model the query points as the system states and the scene points as the system inputs. We design a spatial correlation module to parameterize the SSM based on the initial system states and inputs.
• Figure 4: (a): Illustration of ISSM-based decoder architecture. (b): Detailed Structure of the ISSM-based Bidirectional Scan. (c): Detailed Structure of the Gated Feed-Forward Network (GFFN).
• Figure 5: Effect of the designed modules. We progressively add the proposed modules to the SSM-based baseline to verify the contribution of each module.