Real-time Stereo-based 3D Object Detection for Streaming Perception

TL;DR

This work introduces StreamDSGN, the first real-time stereo-based 3D object detection framework designed for streaming perception that directly predicts the 3D properties of objects in the next moment by leveraging historical information, thereby alleviating the accuracy degradation of streaming perception.

Abstract

The ability to promptly respond to environmental changes is crucial for the perception system of autonomous driving. Recently, a new task called streaming perception was proposed. It jointly evaluate the latency and accuracy into a single metric for video online perception. In this work, we introduce StreamDSGN, the first real-time stereo-based 3D object detection framework designed for streaming perception. StreamDSGN is an end-to-end framework that directly predicts the 3D properties of objects in the next moment by leveraging historical information, thereby alleviating the accuracy degradation of streaming perception. Further, StreamDSGN applies three strategies to enhance the perception accuracy: (1) A feature-flow-based fusion method, which generates a pseudo-next feature at the current moment to address the misalignment issue between feature and ground truth. (2) An extra regression loss for explicit supervision of object motion consistency in consecutive frames. (3) A large kernel backbone with a large receptive field for effectively capturing long-range spatial contextual features caused by changes in object positions. Experiments on the KITTI Tracking dataset show that, compared with the strong baseline, StreamDSGN significantly improves the streaming average precision by up to 4.33%. Our code is available at https://github.com/weiyangdaren/streamDSGN-pytorch.

Figures (8)

• Figure 1: In the context of online streaming perception, the environment changes during inference.
• Figure 2: Illustration of the challenges.
• Figure 3: The architecture of StreamDSGN pipeline. (a) The feature extractor retrieves features from streaming stereo image pairs $\{(\mathbf{I}^{L}_{t}, \mathbf{I}^{R}_{t})|t=1,...,n-1 \}$ and flattens them into BEV features. (b) The depth regression component utilizes $\mathbf{G}^{depth}_{t}$ as the supervision. (c) The BEV detector predicts the object state of the next moment by merging features from the current and previous frames. (d) The Feature-Flow Fusion generates a pseudo-next feature $\mathbf{F}^{pseudo}_{t+1}$ by extrapolating from past features and then concatenates it with the existing historical feature set $\{ \mathbf{F}_{t}, \mathbf{F}_{t-1} \}$.
• Figure 4: A toy example of pseudo-next feature generation.
• Figure 5: Illustration of MCL.