XD-MAP: Cross-Modal Domain Adaptation using Semantic Parametric Mapping

TL;DR

XD-MAP addresses the cross-modal domain adaptation problem by enabling knowledge transfer from camera-based detections to LiDAR without requiring overlapping sensor views. It builds a semantic parametric HD map using SLAM and class-tailored geometric primitives, then renders pseudo labels into the LiDAR domain via a spherical projection, allowing 2D semantic, 2D panoptic, and 3D semantic tasks to benefit from image-domain supervision. The approach yields substantial gains over single-shot baselines across all evaluated tasks (e.g., +$19.5$ mIoU for 2D semantic and +$32.3$ mIoU for 3D semantic) and is robust to ablations on motion compensation, map element range, and sampling frequency. This framework enables scalable, self-supervised cross-sensor knowledge transfer, expanding the utility of rich image datasets for underrepresented sensing modalities and broader 360° perception, with future work on more semantic classes and dynamic objects.

Abstract

Until open-world foundation models match the performance of specialized approaches, the effectiveness of deep learning models remains heavily dependent on dataset availability. Training data must align not only with the target object categories but also with the sensor characteristics and modalities. To bridge the gap between available datasets and deployment domains, domain adaptation strategies are widely used. In this work, we propose a novel approach to transferring sensor-specific knowledge from an image dataset to LiDAR, an entirely different sensing domain. Our method XD-MAP leverages detections from a neural network on camera images to create a semantic parametric map. The map elements are modeled to produce pseudo labels in the target domain without any manual annotation effort. Unlike previous domain transfer approaches, our method does not require direct overlap between sensors and enables extending the angular perception range from a front-view camera to a full 360 view. On our large-scale road feature dataset, XD-MAP outperforms single shot baseline approaches by +19.5 mIoU for 2D semantic segmentation, +19.5 PQth for 2D panoptic segmentation, and +32.3 mIoU in 3D semantic segmentation. The results demonstrate the effectiveness of our approach achieving strong performance on LiDAR data without any manual labeling.

Figures (7)

• Figure 1: Overview of the proposed XD-MAP: We generate pseudo labels from RGB camera images from a single camera using a neural network trained on Mapillary Vistas Neuhold_Mapillary_2017_ICCV, a well-generalizing image dataset, as source. Using an accurate SLAM and parametric geometric representations, tailored to the semantic classes of objects, we build a precise semantic map of objects of interest. The mapped objects can then be rendered into the target domain, in our case a LiDAR sensor, opening up a new sensing modality and extending the perception range from a front-view camera to 360° coverage. As exemplary tasks, we cover 2D panoptic as well as 2D and 3D semantic segmentation.
• Figure 2: Exemplary results of the semantic parametric mapping. Depicted are parametric primitives of three semantic classes (poles, traffic lights, road signs). The parametric map provides pseudo-labels for an accumulated lidar point cloud colored by semantic class and instance, respectively (top row). As sanity check or to transfer between cameras, the map elements can also be projected into camera images (bottom row). Best viewed with digital zoom.
• Figure 3: Illustration of measurement artifacts affecting our pipeline. The parallax effect leads to background lidar points appearing on foreground objects when projected in the camera space.
• Figure 4: Spatial distribution of the sequences in Karlsruhe, Germany. The areas for the test set are depicted in red. The yellow areas show the geographic separation between training and test set.
• Figure 5: Qualitative results of 2D perception.