GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation

TL;DR

GoodSAM++ addresses panoramic semantic segmentation by bridging semantic label gaps and capacity gaps through a Teacher Assistant that enriches SAM-based pseudo supervision. The framework introduces Distortion-Aware Rectification v2 (DARv2) to mitigate ERP distortions and Boundary Enhancement (BEv2), plus Cross-Task Complementary Fusion (CTCFv2) and a Multi-level Knowledge Adaptation (MKA) module to transfer knowledge to a compact student. Extensive outdoor and indoor experiments demonstrate state-of-the-art performance among unsupervised domain adaptation methods, with a remarkably lightweight student (3.7 million parameters) achieving competitive accuracy. The approach shows strong generalization to open-world panoramas and offers a practical path toward label-free, efficient panoramic segmentation.

Abstract

This paper presents GoodSAM++, a novel framework utilizing the powerful zero-shot instance segmentation capability of SAM (i.e., teacher) to learn a compact panoramic semantic segmentation model, i.e., student, without requiring any labeled data. GoodSAM++ addresses two critical challenges: 1) SAM's inability to provide semantic labels and inherent distortion problems of panoramic images; 2) the significant capacity disparity between SAM and the student. The `out-of-the-box' insight of GoodSAM++ is to introduce a teacher assistant (TA) to provide semantic information for SAM, integrated with SAM to obtain reliable pseudo semantic maps to bridge both domain and capacity gaps. To make this possible, we first propose a Distortion-Aware Rectification (DARv2) module to address the domain gap. It effectively mitigates the object deformation and distortion problem in panoramic images to obtain pseudo semantic maps. We then introduce a Multi-level Knowledge Adaptation (MKA) module to efficiently transfer the semantic information from the TA and pseudo semantic maps to our compact student model, addressing the significant capacity gap. We conduct extensive experiments on both outdoor and indoor benchmark datasets, showing that our GoodSAM++ achieves a remarkable performance improvement over the state-of-the-art (SOTA) domain adaptation methods. Moreover, diverse open-world scenarios demonstrate the generalization capacity of our GoodSAM++. Last but not least, our most lightweight student model achieves comparable performance to the SOTA models with only 3.7 million parameters.

Figures (13)

• Figure 1: Performance comparison of GoodSAM++ with GoodSAM and previous SOTA methods zhang2022bendingzheng2023bothzheng2023look across various model parameter ranges. Our GoodSAM++ outperforms our GoodSAM by the largest margin for the tiny size, with a performance gap of 0.77% mIoU.
• Figure 2: (a) Visual Comparisons between GoodSAM and GoodSAM++ on the diverse indoor or outdoor open-world scenes.
• Figure 3: Overview of GoodSAM++ framework, consisting of three models: SAM, teacher assistant, and student. Our method has two main technical components: the Distortion-Aware Rectification (DAR) module and the Multi-level Knowledge Adaptation (MKA) module.
• Figure 4: Overview of the proposed BEv2 block. In blue shaded part, by combining pseudo semantic maps, the confidence map, and the boundary map generated by SAM, we obtain the high and low confidence boundary maps. In the bottom part, it represents the processing of pixels in the high confidence boundary map $B_{E\_H}^i$. Additionally, the green shaded part demonstrates the consideration of whether the pixels in the low confidence boundary map $B_{SAM\_L}^i$ are reliable.
• Figure 5: Example visualization results from the DensePASS test set: (a) Input panorama image, (b) Segformer-B5 xie2021segformer without sliding window sampling, (c) DPPASS-S zheng2023both, (d) DATR-S zheng2023look, (e) GoodSAM-S, (f) GoodSAM++-S, (g) Ground truth.