Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting

TL;DR

This work tackles robust scaling of multi-camera 3D object detection (MC3D-Det) across diverse camera setups and environments by addressing surround refinement degradation caused by monocular-depth overfitting during training. It introduces a weak-to-strong eliciting framework that uses biased, weakly tuned experts and a composite distillation pipeline to fuse large-scale 2D foundation-model knowledge with task-specific cues, plus a dataset-merge strategy to handle varying camera counts and intrinsics. Empirical results on multi-dataset and sim-real benchmarks show improved cross-dataset generalization and real-to-sim transfer for BEV-based detectors, with plug-and-play applicability and no extra inference cost. The approach is validated across nuScenes, Waymo, Lyft, and DeepAccident, and accompanied by an open-source code release.

Abstract

The emergence of Multi-Camera 3D Object Detection (MC3D-Det), facilitated by bird's-eye view (BEV) representation, signifies a notable progression in 3D object detection. Scaling MC3D-Det training effectively accommodates varied camera parameters and urban landscapes, paving the way for the MC3D-Det foundation model. However, the multi-view fusion stage of the MC3D-Det method relies on the ill-posed monocular perception during training rather than surround refinement ability, leading to what we term "surround refinement degradation". To this end, our study presents a weak-to-strong eliciting framework aimed at enhancing surround refinement while maintaining robust monocular perception. Specifically, our framework employs weakly tuned experts trained on distinct subsets, and each is inherently biased toward specific camera configurations and scenarios. These biased experts can learn the perception of monocular degeneration, which can help the multi-view fusion stage to enhance surround refinement abilities. Moreover, a composite distillation strategy is proposed to integrate the universal knowledge of 2D foundation models and task-specific information. Finally, for MC3D-Det joint training, the elaborate dataset merge strategy is designed to solve the problem of inconsistent camera numbers and camera parameters. We set up a multiple dataset joint training benchmark for MC3D-Det and adequately evaluated existing methods. Further, we demonstrate the proposed framework brings a generalized and significant boost over multiple baselines. Our code is at \url{https://github.com/EnVision-Research/Scale-BEV}.

Figures (4)

• Figure 1: Results of different methods on combined datasets. Oracle denotes the results obtained from the evaluations where models are trained and validated on the same dataset.
• Figure 2: Visualization of surround refinement capability. The pink zone (surround refinement ability of $F_{fusion}$) is the disparity between monocular results (red line) and final fusion results (blue line). BEVDet and BEVFormer were jointly trained on nuScenes, Waymo, and Lyft datasets. The Average Precision (AP) results of monocular and fusion outcomes are reported on the Lyft dataset. For brevity, we report only the 'Car' category results in the front view.
• Figure 3: Our joint training framework for the MC3D-Det task. The blue arrows represent the general process of MC3D-Det: different datasets are fed into the monocular perception encoder (central brain $F_{mono}$) to extract preliminary features and then sent to the surround refinement model $F_{fusion}$; The final result is output through the detector. (b) weakly tuned experts distill part of the knowledge from the central brain. These experts replace parts of the sample in the central brain, which is a way of building noise features to augment the fusion module. (c) The composite knowledge of the generalized 2D foundational model, more fine-grained attributes, and standardized virtual depth are comprehensively distilled into the central brain. (d) Strong surround refinement by rendering is used to make the fusion model $F_{fusion}$ learn more robust geometric features.
• Figure 4: PDIR visualization. (a) PDIR reflects how far the ground depth changes over the pixels in the image. (b) PDIR has different distributions in the same dataset, which reflects not only the difference between different data sets but also the difference between specific scenes.