LSSInst: Improving Geometric Modeling in LSS-Based BEV Perception with Instance Representation

TL;DR

This work proposes LSSInst, a two-stage object detector incorporating BEV and instance representations in tandem, which boosts the performances of modern LSS-based BEV perception methods without bells and whistles and outperforms current LSS-based state-of-the-art works on the large-scale nuScenes benchmark.

Abstract

With the attention gained by camera-only 3D object detection in autonomous driving, methods based on Bird-Eye-View (BEV) representation especially derived from the forward view transformation paradigm, i.e., lift-splat-shoot (LSS), have recently seen significant progress. The BEV representation formulated by the frustum based on depth distribution prediction is ideal for learning the road structure and scene layout from multi-view images. However, to retain computational efficiency, the compressed BEV representation such as in resolution and axis is inevitably weak in retaining the individual geometric details, undermining the methodological generality and applicability. With this in mind, to compensate for the missing details and utilize multi-view geometry constraints, we propose LSSInst, a two-stage object detector incorporating BEV and instance representations in tandem. The proposed detector exploits fine-grained pixel-level features that can be flexibly integrated into existing LSS-based BEV networks. Having said that, due to the inherent gap between two representation spaces, we design the instance adaptor for the BEV-to-instance semantic coherence rather than pass the proposal naively. Extensive experiments demonstrated that our proposed framework is of excellent generalization ability and performance, which boosts the performances of modern LSS-based BEV perception methods without bells and whistles and outperforms current LSS-based state-of-the-art works on the large-scale nuScenes benchmark.

Figures (5)

• Figure 1: The conceptual comparison of LSSInst with previous camera-based fashions.
• Figure 2: Overview of LSSInst. The multi-view images with previous $T$ frames are fed into the backbone network for the image features. BEV branch looks around the image feature to generate the BEV feature by view transformation and temporal encoding. Instance adapter aggregates the sparse object-aware feature from the BEV feature and prepares the multiplicate 3D query combination. Instance branch looks back at the image feature and perfects the sparse feature by spatiotemporal sampling and fusion. Lastly, the model makes the final prediction based on the updated output.
• Figure 3: Comparison results of per-classes mAP on nuScenes $\mathtt{val}$ set.
• Figure 4: Qualitative comparison between baseline proposals (red), predictions (blue), their superposition (purple), and GT (white).
• Figure 5: Comparison of LSSInst, the ground truth, and SOLOFusion on nuScenes $\mathtt{val}$ set.