Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillation

TL;DR

This paper tackles unsupervised pixel-wise semantic segmentation in urban driving scenes by leveraging synchronized LiDAR and camera data. It introduces Drive&Segment, a three-stage cross-modal framework that (i) extracts 3D object proposals from LiDAR and projects them to images, (ii) clusters image-features of these segments to form pseudo-classes, and (iii) uses cross-modal distillation with LiDAR-derived spatial constraints to train a transformer-based segmentation model. The approach yields significant, cross-dataset improvements over prior unsupervised methods across Cityscapes, DarkZurich, Nighttime Driving, and ACDC without any manual labels, and demonstrates robustness to challenging conditions. By fusing geometric LiDAR cues with self-supervised image representations, the method advances scalable, annotation-free perception suitable for real-world autonomous driving deployments.

Abstract

This work investigates learning pixel-wise semantic image segmentation in urban scenes without any manual annotation, just from the raw non-curated data collected by cars which, equipped with cameras and LiDAR sensors, drive around a city. Our contributions are threefold. First, we propose a novel method for cross-modal unsupervised learning of semantic image segmentation by leveraging synchronized LiDAR and image data. The key ingredient of our method is the use of an object proposal module that analyzes the LiDAR point cloud to obtain proposals for spatially consistent objects. Second, we show that these 3D object proposals can be aligned with the input images and reliably clustered into semantically meaningful pseudo-classes. Finally, we develop a cross-modal distillation approach that leverages image data partially annotated with the resulting pseudo-classes to train a transformer-based model for image semantic segmentation. We show the generalization capabilities of our method by testing on four different testing datasets (Cityscapes, Dark Zurich, Nighttime Driving and ACDC) without any finetuning, and demonstrate significant improvements compared to the current state of the art on this problem. See project webpage https://vobecant.github.io/DriveAndSegment/ for the code and more.

Figures (14)

• Figure 1: Proposed fully-unsupervised approach. From uncurated images and LiDAR data, our Drive$\&$Segment approach learns a semantic image segmentation model with no manual annotations. The resulting model performs unsupervised semantic segmentation of new unseen datasets without any human labeling. It can segment complex scenes with many objects, including thin structures such as people, bicycles, poles or traffic lights. Black denotes the ignored label.
• Figure 2: Overview of Drive$\&$Segment. We first perform cross-modal segment extraction on training dataset by exploiting raw LiDAR point clouds $\mathcal{P}$ and raw images$\mathcal{I}$. This yields segments $\mathcal{S}^I$ projected onto the image space (§ \ref{['sec:segment_extraction']}). By clustering their self-supervised features, we obtain an unsupervised labeling of these segments (§ \ref{['sec:clustering']}) and, as a consequence, of their pixels. This provides pixel-wise pseudo ground truth for the next learning step. Finally, given the pseudo-labels and the segments, we perform distillation with cross-modal constraints (§ \ref{['sec:distillation']}) that conjugates information of the LiDAR and the images to learn a final segmentation model using a teacher-student architecture. The learnt segmentation model S --highlighted in the figure-- is used for inference on unseen datasets, yielding compelling results (§ \ref{['sec:experiments']}).
• Figure 3: Cross-modal segment extraction. Input raw point cloud (a) is first segmented with bogoslavskyi17pfg into object segment candidates (b), which are then projected into the image (c); Projected segments are densified to get pixel-level pseudo-labels, with missed pixels being labeled as "ignore", as shown in black (d).
• Figure 4: Segment-wise unsupervised pseudo-labeling. First, given object segments $\mathcal{S^I}$ obtained in the segment extraction stage (left), we take crops around all $N$ objects and feed them to a feature extractor to get a set of $N$ feature vectors. Then, we use the $k$-means algorithm to cluster the feature vectors into $k$ clusters. Finally, we assign pixel-wise pseudo-labels to all pixels belonging to each segment based on the corresponding cluster id. Pixels not covered by a segment are assigned the label "ignore" (black).
• Figure 5: Teacher prediction refinement using spatial constraints. First, the teacher $\textsc{T}$ is trained using loss $\mathcal{L}_{\textsc{T}}$ on images in $\mathcal{I}$ together with segmentation maps in $\mathcal{M}$ obtained from segment-wise unsupervised pseudo-labeling. The teacher predictions $\mathrm{\mathbf{Y}}_{\textsc{T} }$ are refined, using LiDAR segments $\mathbf{S}^{P}$, into maps $\mathrm{\hat{\mathbf{Y}}}_{\textsc{T} }$ that are then used to train the student. Note that teacher's predictions span the whole image, producing outputs even in areas where LiDAR segments $\mathbf{S}^P$ are not available.