Object-Scene-Camera Decomposition and Recomposition for Data-Efficient Monocular 3D Object Detection

TL;DR

An online object-scene-camera decomposition and recomposition data manipulation scheme to more efficiently exploit the training data and serve as a plug-and-play component to boost M3OD models, working flexibly with both fully and sparsely supervised settings.

Abstract

Monocular 3D object detection (M3OD) is intrinsically ill-posed, hence training a high-performance deep learning based M3OD model requires a humongous amount of labeled data with complicated visual variation from diverse scenes, variety of objects and camera poses.However, we observe that, due to strong human bias, the three independent entities, i.e., object, scene, and camera pose, are always tightly entangled when an image is captured to construct training data. More specifically, specific 3D objects are always captured in particular scenes with fixed camera poses, and hence lacks necessary diversity. Such tight entanglement induces the challenging issues of insufficient utilization and overfitting to uniform training data. To mitigate this, we propose an online object-scene-camera decomposition and recomposition data manipulation scheme to more efficiently exploit the training data. We first fully decompose training images into textured 3D object point models and background scenes in an efficient computation and storage manner. We then continuously recompose new training images in each epoch by inserting the 3D objects into the freespace of the background scenes, and rendering them with perturbed camera poses from textured 3D point representation. In this way, the refreshed training data in all epochs can cover the full spectrum of independent object, scene, and camera pose combinations. This scheme can serve as a plug-and-play component to boost M3OD models, working flexibly with both fully and sparsely supervised settings. In the sparsely-supervised setting, objects closest to the ego-camera for all instances are sparsely annotated. We then can flexibly increase the annotated objects to control annotation cost. For validation, our method is widely applied to five representative M3OD models and evaluated on both the KITTI and the more complicated Waymo datasets.

Figures (8)

• Figure 1: Implementation of our data manipulation scheme on four released models under both fully and sparsely supervised settings. The $x$-axis represents the percentage of annotated objects used in our scheme out of all objects in the KITTI train split. The $y$-axis represents the moderate level of $AP_{3D}$ in the KITTI validation split. Our method significantly boosts the performance of the base models in both the fully (i.e., 100% annotation ratio) and the sparsely (i.e., 10%, 20%, and 50%) supervised settings. Our method with only 10% annotations even achieves on-par performance with the fully-supervised setting of the first three base models.
• Figure 2: Illustration of our online object-scene-camera decomposition and recomposition data manipulation scheme. We first fully decompose training images into textured 3D object point models and background scenes in an efficient computation and storage manner. We then continuously recompose new training images in each epoch by inserting the 3D objects into the freespace of the background scenes, and rendering them with perturbed camera poses from textured 3D point representation. In this way, the refreshed training data in all epochs can cover the full spectrum of independent object, scene, and camera combinations.
• Figure 3: Object point model rectification. The CAD model is intended to aid visualization.
• Figure 4: Object position resampling. Using a 3D textured point representation, objects can be flexibly resampled to new positions.
• Figure 5: Freespace generation process. For each row of the freespace map in the polar coordinates system, we scan each column to complete the empty area.