Instance-Guided Unsupervised Domain Adaptation for Robotic Semantic Segmentation

TL;DR

This work tackles domain shift in robotic semantic segmentation by enabling deployment-time unsupervised domain adaptation. It combines multi-view consistency from a volumetric 3D map with zero-shot instance segmentation via Segment Anything (SAM) to produce high-quality pseudo-labels, which are then used for self-supervised fine-tuning of the segmentation model. The approach introduces an instance-aware refinement step that uses grid and informed prompting strategies, reporting consistent improvements over a strong multi-view baseline on real-world ScanNet data without target labels. The results demonstrate robust performance gains and reduced sensitivity to 3D reconstruction artifacts, offering a practical path to robust, label-free adaptation in long-running robotic systems.

Abstract

Semantic segmentation networks, which are essential for robotic perception, often suffer from performance degradation when the visual distribution of the deployment environment differs from that of the source dataset on which they were trained. Unsupervised Domain Adaptation (UDA) addresses this challenge by adapting the network to the robot's target environment without external supervision, leveraging the large amounts of data a robot might naturally collect during long-term operation. In such settings, UDA methods can exploit multi-view consistency across the environment's map to fine-tune the model in an unsupervised fashion and mitigate domain shift. However, these approaches remain sensitive to cross-view instance-level inconsistencies. In this work, we propose a method that starts from a volumetric 3D map to generate multi-view consistent pseudo-labels. We then refine these labels using the zero-shot instance segmentation capabilities of a foundation model, enforcing instance-level coherence. The refined annotations serve as supervision for self-supervised fine-tuning, enabling the robot to adapt its perception system at deployment time. Experiments on real-world data demonstrate that our approach consistently improves performance over state-of-the-art UDA baselines based on multi-view consistency, without requiring any ground-truth labels in the target domain.

Figures (5)

• Figure 1: Examples of the improvement provided by our method (third column) to the view--consistent pseudo--labels continual_adaptation_2d3d (middle column).
• Figure 2: Our method (a) first aggregates the source model’s predictions $\mathbf{Y}^*$ into a 3D map to obtain multi--view consistent pseudo--labels $\mathbf{Y}^\texttt{MC}$. (b) These are then refined with a foundation model for instance segmentation. (c) The resulting pseudo--labels $\mathbf{Y}^\texttt{IR}$ are finally used to fine--tune the source model.
• Figure 3: Instance--aware pseudo--labels ($\mathbf{Y}^\texttt{IR}$) generated with grid (left) and informed (right), compared to multi--view consistent ($\mathbf{Y}^\texttt{MC}$) and raw ($\mathbf{Y}^*$) ones. $\mathbf{I}^t$ and $\mathbf{Y}^\texttt{GT}$ denote the RGB images and ground truths.
• Figure 4: Comparison of pseudo--labels from our method using grid ($\mathbf{Y}^\texttt{IR}_\texttt{G}$) and informed ($\mathbf{Y}^\texttt{IR}_\texttt{I}$) prompts, with representative SAM instances and their prompts ($\mathbf{B}_\texttt{G}$, $\mathbf{B}_\texttt{I}$). $\mathbf{I}^t$ and $\mathbf{Y}^\texttt{MC}$ denote the RGB images and multi--view annotations. The first two columns highlight the strengths of grid, while the last two show the improved precision of informed.
• Figure 5: Segmentation improvements with different prompting strategies (IR$_\texttt{G}$, IR$_\texttt{I}$, IR$_\texttt{GI}$) compared to the baselines (MC and PT). The first two rows use a $5\text{cm}$ voxel size, the last two a $3\text{cm}$. Black areas in $\mathbf{Y}^\texttt{GT}$ are due to rendering errors or missing labels; these segmentation errors are improved by our IR.