3D Object Detection from Images for Autonomous Driving: A Survey

TL;DR

This survey addresses image-based 3D object detection for autonomous driving, detailing the problem, data sources, and the rising class of camera-only approaches. It introduces two taxonomies to organize methods: one by 2D versus 3D feature bases, and another by data-, feature-, or result-lifting paradigms, to clarify how 3D results are derived from 2D data. It systematically analyzes pipelines, datasets, evaluation metrics, and architectural components (feature extraction, depth modeling, loss design, and post-processing), and it discusses the role of auxiliary data (LiDAR, stereo, temporal) and pre-training in boosting performance. Finally, it outlines future directions—enhanced depth estimation, multi-modality fusion, temporal and semi/self-supervised learning, and better generalization—to guide research toward practical, robust 3D detection systems.

Abstract

3D object detection from images, one of the fundamental and challenging problems in autonomous driving, has received increasing attention from both industry and academia in recent years. Benefiting from the rapid development of deep learning technologies, image-based 3D detection has achieved remarkable progress. Particularly, more than 200 works have studied this problem from 2015 to 2021, encompassing a broad spectrum of theories, algorithms, and applications. However, to date no recent survey exists to collect and organize this knowledge. In this paper, we fill this gap in the literature and provide the first comprehensive survey of this novel and continuously growing research field, summarizing the most commonly used pipelines for image-based 3D detection and deeply analyzing each of their components. Additionally, we also propose two new taxonomies to organize the state-of-the-art methods into different categories, with the intent of providing a more systematic review of existing methods and facilitating fair comparisons with future works. In retrospect of what has been achieved so far, we also analyze the current challenges in the field and discuss future directions for image-based 3D detection research.

Figures (10)

• Figure 1: Illustration of the monocular 3D object detection task. Given an input image ( left), it aims to predict a 3D bounding box (represented by its location $(x,y,z)$, dimension $(h,w,l)$, and orientation $\theta$ for each object ( middle). We also show the bird's eye view for better visualization ( right).
• Figure 2: The proposed taxonomy for image-based 3D detection. The methods are first divided into '2D feature-based methods' and '3D feature-based methods'. Then we further group them into 'result-lifting-based methods', 'feature-lifting-based methods', and 'data-lifting-based methods'.
• Figure 3: Illustration of the image-based 3D object detection pipelines. We show the data flows of data-lifting, feature-lifting, and result-lifting methods with blue, green, and red arrows respectively.
• Figure 4: Chronological overview of the most relevant image-based 3D detection methods and the profound benchmarks.
• Figure 5: An illustration of the feature lifting methods. Left: 3D features are generated by accumulating 2D features over corresponding areas. Right: image features are weighted by their depth distribution to lift the 2D features into the 3D space. From oftnet and caddn.