Towards Latency-Aware 3D Streaming Perception for Autonomous Driving

TL;DR

This work tackles the latency challenge in 3D perception for autonomous driving by introducing LASP, a Latency-Aware 3D Streaming Perception framework. LASP combines a continuous-time history integration module with a latency-aware predictive detection head, enabling accurate detections despite irregular historical inputs and posterior latency. It introduces a streaming benchmark that evaluates online predictions under varying latency and demonstrates that LASP achieves online performance approaching 80% of offline results on a Jetson AGX Orin without acceleration, while outperforming some acceleration-enhanced baselines. The approach highlights the importance of continuous temporal alignment and trajectory-informed prediction for real-time, latency-resilient 3D perception on edge devices.

Abstract

Although existing 3D perception algorithms have demonstrated significant improvements in performance, their deployment on edge devices continues to encounter critical challenges due to substantial runtime latency. We propose a new benchmark tailored for online evaluation by considering runtime latency. Based on the benchmark, we build a Latency-Aware 3D Streaming Perception (LASP) framework that addresses the latency issue through two primary components: 1) latency-aware history integration, which extends query propagation into a continuous process, ensuring the integration of historical feature regardless of varying latency; 2) latency-aware predictive detection, a module that compensates the detection results with the predicted trajectory and the posterior accessed latency. By incorporating the latency-aware mechanism, our method shows generalization across various latency levels, achieving an online performance that closely aligns with 80\% of its offline evaluation on the Jetson AGX Orin without any acceleration techniques.

Figures (6)

• Figure 1: When deployed on edge devices, random latency introduces two challenges: 1) irregular historical frames, which compromise the effectiveness of discrete transformation; 2) time-lagged prediction, leading to mismatches between the predicted states and real-time detection outcomes.
• Figure 2: The left figure illustrates the runtime distribution on the edge device. We sampled runtime of each frame, applying discrete transformations to process historical feature and directly using time-lagged predictions with delay. The right figure demonstrates the performance degradation compared to offline results, highlighting the limitations of current algorithms in handling historical frames at various intervals and providing online output.
• Figure 3: Our method builds on end-to-end sparse query-based 3D object detectors: 1) We maintain a memory bank containing historical query context embedding and reference centers, which are aligned to current time $t_0$ and fed into the transformer decoder. 2) Assuming that the hidden state follows a motion-aware linear transition over a small time step $\text{d}t$, we integrate it over time to obtain the hidden state at $t_0$. 3) The updated queries are merged with class prior and passed through a intention-guided head to forecast the future trajectories which are re-encoded for further refinement. 4) Data arriving at $t_0$ is processed until $t_1$, generating trajectories that are used to compensate for the movement from $t_1$ to $t_2$.
• Figure 4: Online performance under different streaming frame rate
• Figure 5: Visualization of baseline and our results on nuScenes dataset. We show 3D bboxes predictions in camera images and the bird’s-eye-view.