SGFormer: Spherical Geometry Transformer for 360 Depth Estimation

TL;DR

SGFormer addresses 360-depth estimation under panorama distortion by integrating spherical geometry into a transformer framework. It introduces SPDecoder, which enforces equidistortion, continuity, and surface distance priors via BRP, CR, and CLE, and a query-based GCPE to adapt spatial cues across resolutions. Across Structured3D and Pano3D benchmarks, SGFormer substantially outperforms state-of-the-art methods, including bi-projection fusion and long-range dependency models. The results demonstrate improved global structure perception and sharper depth details, highlighting the practical value for immersive depth estimation.

Abstract

Panoramic distortion poses a significant challenge in 360 depth estimation, particularly pronounced at the north and south poles. Existing methods either adopt a bi-projection fusion strategy to remove distortions or model long-range dependencies to capture global structures, which can result in either unclear structure or insufficient local perception. In this paper, we propose a spherical geometry transformer, named SGFormer, to address the above issues, with an innovative step to integrate spherical geometric priors into vision transformers. To this end, we retarget the transformer decoder to a spherical prior decoder (termed SPDecoder), which endeavors to uphold the integrity of spherical structures during decoding. Concretely, we leverage bipolar re-projection, circular rotation, and curve local embedding to preserve the spherical characteristics of equidistortion, continuity, and surface distance, respectively. Furthermore, we present a query-based global conditional position embedding to compensate for spatial structure at varying resolutions. It not only boosts the global perception of spatial position but also sharpens the depth structure across different patches. Finally, we conduct extensive experiments on popular benchmarks, demonstrating our superiority over state-of-the-art solutions.

Figures (9)

• Figure 1: Brief comparisons between previous typical methods and the proposed method: (a) Bi-projection fusion strategy. Bifuse r1 combines feature maps from equirectangular and cubemap projections. (b) Long-range dependency strategy. EGFormer r7 adapts vertical and horizontal shaped local windows. (c) Our SGFormer presents a query-based global conditional position embedding (GCPE) and integrates spherical geometric priors, significantly improving the performance of panoramic depth estimation.
• Figure 2: Overview of our proposed SGFormer. The SGFormer comprises three major parts: feature extractors for ERP images at varying resolutions, calculation of global conditional position embedding (CGCPE) module, and the SPDecoder that integrates three spherical geometric priors.
• Figure 3: Overview of the query-based CGCPE module. In particular, the extracted feature f3 generates a global key (i.e. GCPE3) by passing through a transformer module with global spherical position embedding (GSPE). The global key is then processed through linear layers to generate keys corresponding to features at various resolutions. The original features of different resolutions are used as queries, which are subsequently operated by $QK^{T}$. After another linear transformation, the corresponding GCPEs at different resolutions are obtained.
• Figure 4: Overview of SPDecoder (left), which is combined with three spherical geometric priors, and SPAttention (right).
• Figure 5: (a) A projection imaging model with the x-axis (North) as the positive direction. (b) A projection imaging model with the y-axis (North) as the positive direction.