Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation

Xu Zheng, Pengyuan Zhou, Athanasios V. Vasilakos, Lin Wang

TL;DR

This work tackles source-free unsupervised domain adaptation for pinhole-to-panoramic semantic segmentation, addressing semantic mismatch, distortion, and style differences between domains. It introduces a multi-projection knowledge extraction strategy using Tangent Projection (TP) and Fixed FoV Projection (FFP), coupled with Panoramic Prototype Adaptation Module (PPAM) and Cross-Dual Attention Module (CDAM) to transfer knowledge to unlabeled panoramic data. The approach optimizes predictions and prototypes with dedicated losses and iteratively refines a global panoramic prototype set, achieving strong gains over existing SFUDA methods across real and synthetic panoramic benchmarks and approaching the performance of methods using source data. This framework advances practical 360-degree perception under privacy constraints and demonstrates the value of projection-aware prototype and feature alignment for panoramic semantic segmentation.

Abstract

This paper addresses an interesting yet challenging problem -- source-free unsupervised domain adaptation (SFUDA) for pinhole-to-panoramic semantic segmentation -- given only a pinhole image-trained model (i.e., source) and unlabeled panoramic images (i.e., target). Tackling this problem is nontrivial due to the semantic mismatches, style discrepancies, and inevitable distortion of panoramic images. To this end, we propose a novel method that utilizes Tangent Projection (TP) as it has less distortion and meanwhile slits the equirectangular projection (ERP) with a fixed FoV to mimic the pinhole images. Both projections are shown effective in extracting knowledge from the source model. However, the distinct projection discrepancies between source and target domains impede the direct knowledge transfer; thus, we propose a panoramic prototype adaptation module (PPAM) to integrate panoramic prototypes from the extracted knowledge for adaptation. We then impose the loss constraints on both predictions and prototypes and propose a cross-dual attention module (CDAM) at the feature level to better align the spatial and channel characteristics across the domains and projections. Both knowledge extraction and transfer processes are synchronously updated to reach the best performance. Extensive experiments on the synthetic and real-world benchmarks, including outdoor and indoor scenarios, demonstrate that our method achieves significantly better performance than prior SFUDA methods for pinhole-to-panoramic adaptation.

Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation

TL;DR

This work tackles source-free unsupervised domain adaptation for pinhole-to-panoramic semantic segmentation, addressing semantic mismatch, distortion, and style differences between domains. It introduces a multi-projection knowledge extraction strategy using Tangent Projection (TP) and Fixed FoV Projection (FFP), coupled with Panoramic Prototype Adaptation Module (PPAM) and Cross-Dual Attention Module (CDAM) to transfer knowledge to unlabeled panoramic data. The approach optimizes predictions and prototypes with dedicated losses and iteratively refines a global panoramic prototype set, achieving strong gains over existing SFUDA methods across real and synthetic panoramic benchmarks and approaching the performance of methods using source data. This framework advances practical 360-degree perception under privacy constraints and demonstrates the value of projection-aware prototype and feature alignment for panoramic semantic segmentation.

Abstract

This paper addresses an interesting yet challenging problem -- source-free unsupervised domain adaptation (SFUDA) for pinhole-to-panoramic semantic segmentation -- given only a pinhole image-trained model (i.e., source) and unlabeled panoramic images (i.e., target). Tackling this problem is nontrivial due to the semantic mismatches, style discrepancies, and inevitable distortion of panoramic images. To this end, we propose a novel method that utilizes Tangent Projection (TP) as it has less distortion and meanwhile slits the equirectangular projection (ERP) with a fixed FoV to mimic the pinhole images. Both projections are shown effective in extracting knowledge from the source model. However, the distinct projection discrepancies between source and target domains impede the direct knowledge transfer; thus, we propose a panoramic prototype adaptation module (PPAM) to integrate panoramic prototypes from the extracted knowledge for adaptation. We then impose the loss constraints on both predictions and prototypes and propose a cross-dual attention module (CDAM) at the feature level to better align the spatial and channel characteristics across the domains and projections. Both knowledge extraction and transfer processes are synchronously updated to reach the best performance. Extensive experiments on the synthetic and real-world benchmarks, including outdoor and indoor scenarios, demonstrate that our method achieves significantly better performance than prior SFUDA methods for pinhole-to-panoramic adaptation.
Paper Structure (14 sections, 12 equations, 5 figures, 7 tables)

This paper contains 14 sections, 12 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Source-free UDA for Segmentation
  4. UDA for Panoramic Semantic Segmentation
  5. Methodology
  6. Overview
  7. Knowledge Extraction
  8. Knowledge Adaptation
  9. Optimization
  10. Experiments and Analysis
  11. Datasets and Implementation Details.
  12. Experimental Results.
  13. Ablation Study
  14. Conclusion

Figures (5)

  • Figure 1: We address a new problem of achieving source-free pinhole-to-panoramic adaptation for segmentation.
  • Figure 2: Overall framework of our proposed SFUDA for panoramic semantic segmentation.
  • Figure 3: Illustration of the prototype extraction (PE) in the panoramic prototype adaptation module (PPAM).
  • Figure 4: Example visualization results. (a) source, (b) SFDA liu2021source, (c) DATC yang2022source, (d) Ours, (e) Ground Truth (GT).
  • Figure 5: TSNE visualization of (a) Cityscapes-to-DensePASS and (b) SynPASS-to-DensePASS.