Orthographic Feature Transform for Monocular 3D Object Detection

TL;DR

This work tackles monocular 3D object detection by translating image features into a consistent birds-eye-view (BEV) representation using the Orthographic Feature Transform (OFT). By performing reasoning in BEV with a topdown network and predicting BEV confidence maps and bounding-box offsets, the method avoids depth-induced distortions inherent to image space. The approach is end-to-end trainable and leverages fast integral-image pooling to handle a large BEV region set. Evaluated on KITTI, OFT-Net achieves state-of-the-art results among monocular methods and demonstrates strong distance robustness and qualitative BEV localization advantages, highlighting the practical potential for monocular 3D perception in autonomous driving.

Abstract

3D object detection from monocular images has proven to be an enormously challenging task, with the performance of leading systems not yet achieving even 10\% of that of LiDAR-based counterparts. One explanation for this performance gap is that existing systems are entirely at the mercy of the perspective image-based representation, in which the appearance and scale of objects varies drastically with depth and meaningful distances are difficult to infer. In this work we argue that the ability to reason about the world in 3D is an essential element of the 3D object detection task. To this end, we introduce the orthographic feature transform, which enables us to escape the image domain by mapping image-based features into an orthographic 3D space. This allows us to reason holistically about the spatial configuration of the scene in a domain where scale is consistent and distances between objects are meaningful. We apply this transformation as part of an end-to-end deep learning architecture and achieve state-of-the-art performance on the KITTI 3D object benchmark.\footnote{We will release full source code and pretrained models upon acceptance of this manuscript for publication.

Figures (5)

• Figure 1: 3D bounding box detection from monocular images. The proposed system maps image-based features to an orthographic birds-eye-view and predicts confidence maps and bounding box offsets in this space. These outputs are then decoded via non-maximum suppression to yield discrete bounding box predictions.
• Figure 2: Orthographic Feature Transform (OFT). Voxel-based features $\mathbf{g}(x, y, z)$ are generated by accumulating image-based features $\mathbf{f}(u, v)$ over the projected voxel area. The voxel features are then collapsed along the vertical dimension to yield orthographic ground plane features $\mathbf{h}(x, z)$.
• Figure 3: Architecture overview. A front-end ResNet feature extractor generates image-based features, which are mapped to an orthographic representation via our proposed orthographic feature transform. The topdown network processes these features in the birds-eye-view space and at each location on the ground plane predicts a confidence score $S$, a position offset $\boldsymbol{\Delta_{pos}}$, a dimension offset $\boldsymbol{\Delta_{dim}}$ and an angle vector $\boldsymbol{\Delta_{ang}}$.
• Figure 4: Ablation study showing the effect of reducing the number of layers in the topdown network on performance for two different frontend architectures. Zero layers implies that topdown network has been removed entirely.
• Figure 5: Average BEV precision (val) as a function of the minimum distance of objects from the camera. We use an IoU threshold of 0.5 to better compare performance at large depths.