DualCross: Cross-Modality Cross-Domain Adaptation for Monocular BEV Perception

TL;DR

This work proposes DualCross, a cross-modality cross-domain adaptation framework to facilitate the learning of a more robust monocular bird's-eye-view (BEV) perception model, which transfers the point cloud knowledge from a LiDAR sensor in one domain during the training phase to the camera-only testing scenario in a different domain.

Abstract

Closing the domain gap between training and deployment and incorporating multiple sensor modalities are two challenging yet critical topics for self-driving. Existing work only focuses on single one of the above topics, overlooking the simultaneous domain and modality shift which pervasively exists in real-world scenarios. A model trained with multi-sensor data collected in Europe may need to run in Asia with a subset of input sensors available. In this work, we propose DualCross, a cross-modality cross-domain adaptation framework to facilitate the learning of a more robust monocular bird's-eye-view (BEV) perception model, which transfers the point cloud knowledge from a LiDAR sensor in one domain during the training phase to the camera-only testing scenario in a different domain. This work results in the first open analysis of cross-domain cross-sensor perception and adaptation for monocular 3D tasks in the wild. We benchmark our approach on large-scale datasets under a wide range of domain shifts and show state-of-the-art results against various baselines.

Figures (5)

• Figure 1: Left & Middle: Existing adaptation models assume either a fixed modality or a fixed domain between training and testing phases. Right: A more realistic setting considers both cross-modality and cross-domain shifts. We propose DualCross to reduce the domain and modality discrepancy, and achieve state-of-the-art performance.
• Figure 2: Overview of our DualCross framework. DualCross includes three components. (1) LiDAR-Teacher uses voxelized LiDAR point clouds to transform the image features to the BEV frame. It provides essential knowledge on how to guide image learning given LiDAR information. (2) Camera-Student is supervised by the teacher model as well as the LiDAR ground truth. (3) Discriminators are used to align features from source and target domains.
• Figure 3: Qualitative Results in Day $\mathbf{\rightarrow}$ Night setting (model is trained with daytime data, and validated with night data). We notice that DualCross performs significantly better than other baselines for vehicles, drivable roads, and lane marking classes. From left to right: (1) Vanilla adversarial learning; (2) LiDAR as depth supervision with adversarial learning; (3) our DualCross model; (4) Ground Truth. Best viewed in color.
• Figure 4: Results of DualCross improve as the number of LiDAR points increases.
• Figure 5: Visualization of failure cases of DualCross (model is trained with daytime data, and validated with night data). Up: Input images. Left: Prediction from DualCross. Right: Ground truth. We notice that major failure cases of DualCross are far distance and occlusions of objects and regions.