REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion

TL;DR

Panoramic Semantic Segmentation (PASS) seeks pixel-level understanding over 360° imagery, but many methods underutilize panoramic geometry by relying on ERP with RGB or using depth in conventional formats. This work REL-SF4PASS introduces a 3-channel REL depth representation in cylindrical coordinates (Rectified Depth, Elevation-Gained Vertical Inclination Angle, and Lateral Orientation Angle) and a Spherical-dynamic Multi-Modal Fusion (SMMF) to fuse RGB and REL features regionally on the cylinder surface. The approach yields clear accuracy and robustness gains on the Stanford2D3D Panoramic datasets, with notable improvements across folds and reduced sensitivity to 3D disturbances. By exploiting spherical geometry priors and region-adaptive fusion, the method enhances both depth interpretation and multi-modal integration for PASS.

Abstract

As an important and challenging problem in computer vision, Panoramic Semantic Segmentation (PASS) aims to give complete scene perception based on an ultra-wide angle of view. Most PASS methods often focus on spherical geometry with RGB input or using the depth information in original or HHA format, which does not make full use of panoramic image geometry. To address these shortcomings, we propose REL-SF4PASS with our REL depth representation based on cylindrical coordinate and Spherical-dynamic Multi-Modal Fusion SMMF. REL is made up of Rectified Depth, Elevation-Gained Vertical Inclination Angle, and Lateral Orientation Angle, which fully represents 3D space in cylindrical coordinate style and the surface normal direction. SMMF aims to ensure the diversity of fusion for different panoramic image regions and reduce the breakage of cylinder side surface expansion in ERP projection, which uses different fusion strategies to match the different regions in panoramic images. Experimental results show that REL-SF4PASS considerably improves performance and robustness on popular benchmark, Stanford2D3D Panoramic datasets. It gains 2.35% average mIoU improvement on all 3 folds and reduces the performance variance by approximately 70% when facing 3D disturbance.

Figures (7)

• Figure 1: Overview of ERP projection, cylindrical coordinate and comparison of HHA and REL. $\hat{T}_{\theta, p}$ is the tangent line to circle $O_p$ at point $p$. Circle $O_p$ lies in the plane perpendicular to gravity, passes through point $p$, and its center lies on the cylindrical axis $L_{cyl}$ of $\rho \theta z$. In (c), we compared how different structure information are represented in REL and HHA.
• Figure 2: The overview of REL-SF4PASS. Firstly, REL representation effectively represents the depth information by using ReD, EGVIA, and LOA, which contains the 3D location and surface normal direction. Secondly, Our SMMF to get image regions from cylinder side surface and uses a Gate Network to independently determine the fusion used in each region.
• Figure 3: The calculation process (a) and physical model (b) of REL representation. In (a), $\mathbf{R}$ is 3D Rigid Body Rotation, and $\oplus$ is \ref{['eq:Ecal']}.
• Figure 4: The detail design of SMMF Region Slicing (a) and Gate Network (b). Slicings have overlap and $\oplus$ means the fusion operation.
• Figure 5: Comparison of normalized height and the angle between surface normal and gravity direction (short as Angle). The x-axis of the right figure is $\phi$ (from $-90^{\circ}$ to $90^{\circ}$) when the y one is the average of all normalized corresponding values.