WidthFormer: Toward Efficient Transformer-based BEV View Transformation

TL;DR

WidthFormer tackles real-time BEV view transformation from multi-view cameras for autonomous driving by introducing a single-decoder-layer transformer module that operates on vertically compressed width features. It is powered by Reference Positional Encoding (RefPE), which encodes 3D geometry for both BEV transformation and sparse 3D detectors, and is complemented by a Refine Transformer and auxiliary training tasks to compensate for information loss due to feature compression. The approach achieves state-of-the-art efficiency-accuracy trade-offs on nuScenes, with millisecond-scale latencies on consumer hardware and strong robustness to camera perturbations, making it suitable for edge deployment. The work also provides practical insights and code to facilitate deployment in real-world, complex driving environments.

Abstract

We present WidthFormer, a novel transformer-based module to compute Bird's-Eye-View (BEV) representations from multi-view cameras for real-time autonomous-driving applications. WidthFormer is computationally efficient, robust and does not require any special engineering effort to deploy. We first introduce a novel 3D positional encoding mechanism capable of accurately encapsulating 3D geometric information, which enables our model to compute high-quality BEV representations with only a single transformer decoder layer. This mechanism is also beneficial for existing sparse 3D object detectors. Inspired by the recently proposed works, we further improve our model's efficiency by vertically compressing the image features when serving as attention keys and values, and then we develop two modules to compensate for potential information loss due to feature compression. Experimental evaluation on the widely-used nuScenes 3D object detection benchmark demonstrates that our method outperforms previous approaches across different 3D detection architectures. More importantly, our model is highly efficient. For example, when using $256\times 704$ input images, it achieves 1.5 ms and 2.8 ms latency on NVIDIA 3090 GPU and Horizon Journey-5 computation solutions. Furthermore, WidthFormer also exhibits strong robustness to different degrees of camera perturbations. Our study offers valuable insights into the deployment of BEV transformation methods in real-world, complex road environments. Code is available at https://github.com/ChenhongyiYang/WidthFormer .

Figures (6)

• Figure 1: Comparison on different BEV view transformation paradigms.
• Figure 2: Reference Positional Encoding (RefPE): RefPE has a rotation and a distance part. For sparse 3D detectors, it has another height part. The rotation encoding is computed using a camera ray's rotation degree on the BEV plane. To compute the point-wise distance PE and height PE, we leverage the reference coefficients, predicted from the visual features, to aggregate the distance & height PEs of reference points on a camera ray. We remove the height PE for width features and compute their distance PE by aggregating all point-wise distance PE along an image column using a predicted height distribution.
• Figure 3: WidthFormer takes multi-view images as input and outputs the converted BEV features. It first compresses the image features into width features by pooling their height dimension. Then the width features are refined with a Refine Transformer to compensate for potential information loss. After adding with our Reference Positional Encoding, the width features are fed into the transformer decoder to serve as keys & values, which are queried by the BEV query vectors computed from the pre-defined BEV polar coordinates.
• Figure 4: CUDA Latency and mAP trade-off comparison of different VT methods using BEVDet on various size settings.
• Figure 5: Latency comparison of different VT methods on the Horizon Journey-5 computation solutions.