MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection

TL;DR

MonoPlace3D tackles the data-hungry nature of monocular 3D object detection by learning a scene-conditioned distribution over plausible 3D bounding boxes and using it to generate realistic augmentations. The approach comprises SA-PlaceNet, which combines geometry-aware augmentation with a probabilistic box representation, and a rendering pipeline that produces highly realistic cars via ShapeNet assets and ControlNet-based diffusion. Across KITTI and NuScenes, the method yields consistent gains over strong baselines and improves data efficiency, reducing the need for large real datasets. The framework is modular, scalable to additional categories, and demonstrates practical impact for outdoor autonomous perception, albeit with dependencies on inpainting quality and lighting modeling.

Abstract

Current monocular 3D detectors are held back by the limited diversity and scale of real-world datasets. While data augmentation certainly helps, it's particularly difficult to generate realistic scene-aware augmented data for outdoor settings. Most current approaches to synthetic data generation focus on realistic object appearance through improved rendering techniques. However, we show that where and how objects are positioned is just as crucial for training effective 3D monocular detectors. The key obstacle lies in automatically determining realistic object placement parameters - including position, dimensions, and directional alignment when introducing synthetic objects into actual scenes. To address this, we introduce MonoPlace3D, a novel system that considers the 3D scene content to create realistic augmentations. Specifically, given a background scene, MonoPlace3D learns a distribution over plausible 3D bounding boxes. Subsequently, we render realistic objects and place them according to the locations sampled from the learned distribution. Our comprehensive evaluation on two standard datasets KITTI and NuScenes, demonstrates that MonoPlace3D significantly improves the accuracy of multiple existing monocular 3D detectors while being highly data efficient.

Figures (18)

• Figure 1: a) We compare augmentations from our learned placement with heuristic-based placements from Lift3D li2023lift3d. In our augmentations, vehicles follow the lane orientations and are placed appropriately. b) These realistic augmentations significantly improve the 3D detection performance (KITTI kitti_chen_split val set, (easy)). Notably, we achieve detection performance comparable to that of the fully labeled dataset using only $50\%$ of the dataset.
• Figure 2: a) SA-PlaceNet Architecture: Given an input background image and corresponding depth to predict the means of a multi-dimensional Gaussian distribution over 3D bounding boxes. 3D bounding boxes are sampled from each of these Gaussian to compute the training loss. b) Geometry-aware augmentation in BEV (Birds Eye View). For a given source car location ($b_{loc}$), we first find $K$ nearest neighbors with the same orientation and augment the location to $\tilde{b}_{loc}$ by interpolating with neighboring locations $n_{loc}$ (Alg.\ref{['alg:alg1']})
• Figure 3: Rendering pipeline: Given a 3D asset, we first render an image and shadow from a fixed light source according to the 3D box parameters. Next, we used edge-conditioned ControlNet controlnet to generate a realistic car version that follows the same orientation and scale as the rendered image. Finally, we use the obtained shadow, rendered car, and 3D location to place the car and render augmented images.
• Figure 4: Given an input source image, we plot the heatmaps of the mean objectness score at each pixel location. The generated heatmaps span a large region on the road with plausible locations of objects. Next, we show samples of bounding boxes and realistic renderings of cars in the scene.
• Figure 5: a) Ablation for object placement - For a background road scene, we visualize the heatmaps of aggregated objectness scores at each pixel location. Geometric augmentation and variational inference help to generate diverse and plausible object placements. b) Histogram of the distribution of orientations of the ground truth bounding boxes and the generated bounding boxes.