StereoDETR: Stereo-based Transformer for 3D Object Detection

TL;DR

StereoDETR tackles the speed-accuracy trade-off in camera-based 3D detection by delivering a concise, end-to-end Transformer framework that fuses a lightweight monocular DETR main branch with a stereo depth branch through differentiable depth sampling. A depth-sampling mechanism addresses occlusion, while auxiliary depth and disparity supervision align depth cues with object predictions, enabling real-time inference at 56.82 FPS (17.6 ms) with competitive $AP_{3D}$ and $AP_{BEV}$ on KITTI. The approach achieves state-of-the-art real-time performance for pedestrians and cyclists, and reduces computational cost to 59.80 GFLOPs with ~18.4M parameters, roughly half of MonoDETR and far smaller than other stereo baselines. This work demonstrates that end-to-end stereo 3D detection can outperform monocular detectors in speed without sacrificing accuracy, offering a practical path toward efficient open-world 3D perception.

Abstract

Compared to monocular 3D object detection, stereo-based 3D methods offer significantly higher accuracy but still suffer from high computational overhead and latency. The state-of-the-art stereo 3D detection method achieves twice the accuracy of monocular approaches, yet its inference speed is only half as fast. In this paper, we propose StereoDETR, an efficient stereo 3D object detection framework based on DETR. StereoDETR consists of two branches: a monocular DETR branch and a stereo branch. The DETR branch is built upon 2D DETR with additional channels for predicting object scale, orientation, and sampling points. The stereo branch leverages low-cost multi-scale disparity features to predict object-level depth maps. These two branches are coupled solely through a differentiable depth sampling strategy. To handle occlusion, we introduce a constrained supervision strategy for sampling points without requiring extra annotations. StereoDETR achieves real-time inference and is the first stereo-based method to surpass monocular approaches in speed. It also achieves competitive accuracy on the public KITTI benchmark, setting new state-of-the-art results on pedestrian and cyclist subsets. The code is available at https://github.com/shiyi-mu/StereoDETR-OPEN.

Figures (8)

• Figure 1: Comparison of accuracy and speed with existing camera-based methods on the KITTI test set (Car category, moderate difficulty).
• Figure 2: Comparison with existing monocular and stereo architectures. Based on the monocular 2D framework DETR, we introduce a simple extension by incorporating multi-scale correlation computation for depth map estimation, and propose an offset-based depth sampling strategy to address occlusion issues.
• Figure 3: Overview of StereoDETR. The framework consists of two branches: DETR branch and stereo branch. The DETR branch includes a visual encoder-decoder and detection heads to predict object class, orientation, size, 3D center, and depth sampling points. The stereo branch computes a correlation volume, applies Multi-Scale Fusion (MSF), and predicts the depth map and disparity map. The two branches are connected via a Grid Sampling module.
• Figure 4: Multi-Scale Fusion module: transforms multi-scale correlation volumes into depth features.
• Figure 5: Depth map decoder and disparity decoder modules