Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Jiaming Zhang, Kailun Yang, Hao Shi, Simon Reiß, Kunyu Peng, Chaoxiang Ma, Haodong Fu, Philip H. S. Torr, Kaiwei Wang, Rainer Stiefelhagen

TL;DR

The paper introduces Trans4PASS+, a distortion-aware Transformer framework for panoramic semantic segmentation, featuring Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) to counteract equirectangular projection distortions. It couples this architecture with Mutual Prototypical Adaptation (MPA) and SAM-based pseudo-label rectification to enable effective unsupervised domain adaptation across Pin2Pan and Syn2Real paradigms, and it introduces SynPASS, a synthetic panoramic dataset for synthetic-to-real transfer. Across indoor and outdoor benchmarks, Trans4PASS+ achieves state-of-the-art performance on four panoramic segmentation datasets and demonstrates strong zero-shot and few-shot generalization relative to fully supervised baselines. The work provides a practical pathway to robust 360° scene understanding, enabling better autonomous navigation, robotics, and immersive visualization in real-world environments.

Abstract

In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360° imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360° imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

TL;DR

The paper introduces Trans4PASS+, a distortion-aware Transformer framework for panoramic semantic segmentation, featuring Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) to counteract equirectangular projection distortions. It couples this architecture with Mutual Prototypical Adaptation (MPA) and SAM-based pseudo-label rectification to enable effective unsupervised domain adaptation across Pin2Pan and Syn2Real paradigms, and it introduces SynPASS, a synthetic panoramic dataset for synthetic-to-real transfer. Across indoor and outdoor benchmarks, Trans4PASS+ achieves state-of-the-art performance on four panoramic segmentation datasets and demonstrates strong zero-shot and few-shot generalization relative to fully supervised baselines. The work provides a practical pathway to robust 360° scene understanding, enabling better autonomous navigation, robotics, and immersive visualization in real-world environments.

Abstract

In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360° imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360° imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.
Paper Structure (32 sections, 17 equations, 36 figures, 15 tables)

This paper contains 32 sections, 17 equations, 36 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Segmentation
  4. Panoramic Segmentation
  5. Dynamic and Deformable Vision Transformers
  6. Unsupervised Domain Adaptation
  7. Methodology
  8. Trans4PASS+ Architecture
  9. Deformable Patch Embedding
  10. Deformable MLP
  11. Mutual Prototypical Adaptation
  12. SynPASS: Proposed Synthetic Dataset
  13. Experiments
  14. Datasets and Settings
  15. SynPASS Benchmark
  16. ...and 17 more sections

Figures (36)

  • Figure 1: Performance of Pin2Pan
  • Figure 2: Performance of Syn2Real
  • Figure 4: Performance gains of Trans4PASS+ in the settings of Source-Only (SO) and panoramic Unsupervised Domain Adaptation (UDA), transferring from Stanford2D3D Pinhole to Panoramic (SPinSPan) and from Cityscapes to DensePASS (CSDP) domains.
  • Figure 5: Domain adaptation paradigms (Pin2Pan and Syn2Real)
  • Figure 6: Distribution sidewalk
  • ...and 31 more figures