RAFT -- A Domain Adaptation Framework for RGB & LiDAR Semantic Segmentation

TL;DR

RAFT addresses the Syn2Real gap in RGB and LiDAR semantic segmentation by unifying active domain adaptation with hyperbolic feature augmentation and modality-specific domain mixing. It extends HALO and Hyperbolic Feature Augmentation to dense prediction, leveraging hyperbolic geometry for uncertainty-aware sampling and per-class feature synthesis, while employing DACS for RGB and PolarMix for LiDAR to align cross-domain representations. Across standard Syn2Real benchmarks and real-world Go2 validation, RAFT achieves state-of-the-art gains with a small annotation budget, demonstrating robust generalization and practical deployment potential. An open Livox Mid-360 LiDAR dataset further supports continued research in real-world domain adaptation for robotics.

Abstract

Image segmentation is a powerful computer vision technique for scene understanding. However, real-world deployment is stymied by the need for high-quality, meticulously labeled datasets. Synthetic data provides high-quality labels while reducing the need for manual data collection and annotation. However, deep neural networks trained on synthetic data often face the Syn2Real problem, leading to poor performance in real-world deployments. To mitigate the aforementioned gap in image segmentation, we propose RAFT, a novel framework for adapting image segmentation models using minimal labeled real-world data through data and feature augmentations, as well as active learning. To validate RAFT, we perform experiments on the synthetic-to-real "SYNTHIA->Cityscapes" and "GTAV->Cityscapes" benchmarks. We managed to surpass the previous state of the art, HALO. SYNTHIA->Cityscapes experiences an improvement in mIoU* upon domain adaptation of 2.1%/79.9%, and GTAV->Cityscapes experiences a 0.4%/78.2% improvement in mIoU. Furthermore, we test our approach on the real-to-real benchmark of "Cityscapes->ACDC", and again surpass HALO, with a gain in mIoU upon adaptation of 1.3%/73.2%. Finally, we examine the effect of the allocated annotation budget and various components of RAFT upon the final transfer mIoU.

Figures (3)

• Figure -1: The effect on mIoU and mIoU* of applying various RAFT components in performing domain adaptation of a SegFormer B0 model from SYNTHIA to Cityscapes. The mIoU* metric uses 13 common classes in both SYNTHIA and Cityscapes, while the mIoU metric uses all 16 classes shared between SYNTHIA and Cityscapes. Partial RAFT A includes HALO along with HFA and hyperbolic mixup, Partial RAFT B includes the aforementioned components plus the focal loss, and RAFT includes all RAFT components.
• Figure 0: The Input row contains the raw RGB and LiDAR scans collected from our Unitree Go2 Edu robot. The Baseline ADA row displays the segmentation masks created by the baseline ADA-only adapted models. Finally, the RAFT row displays the segmentation masks created by our RAFT-adapted models. Due to the sparsity of the LiDAR point clouds, we magnify the two regions which display improvements with the RAFT-trained model.
• Figure 1: The quantitative mIoU results of applying RAFT and Annotator with varying voxel budgets for active learning. While RAFT initially performs worse than Annotator; with the addition of only one more voxel for active learning, it not only outperforms Annotator, but comes close to matching the performance of the model trained on a mix of source SynLiDAR and target Livox Mid-360 LiDAR scans.