Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift

Björn Michele, Alexandre Boulch, Gilles Puy, Tuan-Hung Vu, Renaud Marlet, Nicolas Courty

TL;DR

The paper addresses domain shift in lidar-based 3D semantic segmentation and uses unsupervised multimodal distillation from vision foundation models to improve cross-domain generalization. Its MuDDoS pipeline distills features from a frozen VFM-backed backbone via image-to-lidar distillation, trains an MLP classification head on source labels, and applies self-training for refinement. Key findings show that backbone architecture and normalization layers critically affect generalization, that DINOv2-based distillation outperforms SAM, and that pretraining on multiple datasets enables a single backbone to address many domain shifts, sometimes surpassing state-of-the-art without target labels. The approach yields state-of-the-art results, with improvements up to $24.2 mIoU$ points on challenging cross-domain pairs, and the authors release the code for reproducibility.

Abstract

Semantic segmentation networks trained under full supervision for one type of lidar fail to generalize to unseen lidars without intervention. To reduce the performance gap under domain shifts, a recent trend is to leverage vision foundation models (VFMs) providing robust features across domains. In this work, we conduct an exhaustive study to identify recipes for exploiting VFMs in unsupervised domain adaptation for semantic segmentation of lidar point clouds. Building upon unsupervised image-to-lidar knowledge distillation, our study reveals that: (1) the architecture of the lidar backbone is key to maximize the generalization performance on a target domain; (2) it is possible to pretrain a single backbone once and for all, and use it to address many domain shifts; (3) best results are obtained by keeping the pretrained backbone frozen and training an MLP head for semantic segmentation. The resulting pipeline achieves state-of-the-art results in four widely-recognized and challenging settings. The code will be available at: https://github.com/valeoai/muddos.

Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift

TL;DR

The paper addresses domain shift in lidar-based 3D semantic segmentation and uses unsupervised multimodal distillation from vision foundation models to improve cross-domain generalization. Its MuDDoS pipeline distills features from a frozen VFM-backed backbone via image-to-lidar distillation, trains an MLP classification head on source labels, and applies self-training for refinement. Key findings show that backbone architecture and normalization layers critically affect generalization, that DINOv2-based distillation outperforms SAM, and that pretraining on multiple datasets enables a single backbone to address many domain shifts, sometimes surpassing state-of-the-art without target labels. The approach yields state-of-the-art results, with improvements up to points on challenging cross-domain pairs, and the authors release the code for reproducibility.

Abstract

Semantic segmentation networks trained under full supervision for one type of lidar fail to generalize to unseen lidars without intervention. To reduce the performance gap under domain shifts, a recent trend is to leverage vision foundation models (VFMs) providing robust features across domains. In this work, we conduct an exhaustive study to identify recipes for exploiting VFMs in unsupervised domain adaptation for semantic segmentation of lidar point clouds. Building upon unsupervised image-to-lidar knowledge distillation, our study reveals that: (1) the architecture of the lidar backbone is key to maximize the generalization performance on a target domain; (2) it is possible to pretrain a single backbone once and for all, and use it to address many domain shifts; (3) best results are obtained by keeping the pretrained backbone frozen and training an MLP head for semantic segmentation. The resulting pipeline achieves state-of-the-art results in four widely-recognized and challenging settings. The code will be available at: https://github.com/valeoai/muddos.

Paper Structure

This paper contains 32 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Monomodal UDA for semantic segmentation.
  4. Multimodal UDA for semantic segmentation.
  5. Image-to-lidar distillation.
  6. Study: How to get the best out of VFMs for 3D UDA?
  7. Experimental setup.
  8. Choice of lidar backbone architecture
  9. Effect of intensity.
  10. Effect of normalization layers.
  11. Effect of backbone and capacity.
  12. Choice of a visual foundation model for pretraining
  13. Pretraining protocol.
  14. SAM vs DINOv2.
  15. Downstream training
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Overview of the multimodal distillation pipeline for 3D domain adaptation. With MuDDoS, adapting from an annotated source dataset to an unannotated target dataset, operating in three steps. Step 1 is a 2D-to-3D distillation using a frozen visual foundation model (DINOv2) to obtain aligned 3D representations on all datasets. Step 2 trains a classification head with source labels. The backbone is frozen to prevent the 3D representations from drifting away and to maintain a good performance on the target dataset. Step 3 is a prediction refinement using self-training obtained via a classical teacher-student scheme.
  • Figure 2: Qualitative results on N$\mathrel{ \mkern2mu \clipbox{{.5} 0 0 0}{$$} }$K (Top) and N$\mathrel{ \mkern2mu \clipbox{{.5} 0 0 0}{$$} }$W (Bottom). The label colors correspond to ground truth label assigned color. Points with a ground-truth not belonging to the shown class are grayed out. The source only model tends to over predict vegetation and sometimes mistakes dense partially occluded object with other classes, e.g., pedestrian instead of motorcycle in the second example. MuDDoS is able to partially or completely recover the correct classes.