Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion

TL;DR

Elite360D tackles 360° depth estimation by addressing ERP distortions and limited global context with a dual-projection framework. It introduces ICOSAP as a spatially continuous, global-proceptive point-set projection and fuses ERP and ICOSAP features through a Bi-Projection Bi-Attention Fusion (B2F) that jointly models semantic- and distance-aware dependencies. The method supports diverse ERP backbones and uses a compact ICOSAP encoder to keep parameters near 1M, while achieving competitive or superior results against state-of-the-art bi-projection methods on Matterport3D, Stanford2D3D, and Structured3D. This approach demonstrates that incorporating non-Euclidean projections and cross-attention fusion can yield significant accuracy gains with low computational overhead in 360° perception tasks.

Abstract

360 depth estimation has recently received great attention for 3D reconstruction owing to its omnidirectional field of view (FoV). Recent approaches are predominantly focused on cross-projection fusion with geometry-based re-projection: they fuse 360 images with equirectangular projection (ERP) and another projection type, e.g., cubemap projection to estimate depth with the ERP format. However, these methods suffer from 1) limited local receptive fields, making it hardly possible to capture large FoV scenes, and 2) prohibitive computational cost, caused by the complex cross-projection fusion module design. In this paper, we propose Elite360D, a novel framework that inputs the ERP image and icosahedron projection (ICOSAP) point set, which is undistorted and spatially continuous. Elite360D is superior in its capacity in learning a representation from a local-with-global perspective. With a flexible ERP image encoder, it includes an ICOSAP point encoder, and a Bi-projection Bi-attention Fusion (B2F) module (totally ~1M parameters). Specifically, the ERP image encoder can take various perspective image-trained backbones (e.g., ResNet, Transformer) to extract local features. The point encoder extracts the global features from the ICOSAP. Then, the B2F module captures the semantic- and distance-aware dependencies between each pixel of the ERP feature and the entire ICOSAP feature set. Without specific backbone design and obvious computational cost increase, Elite360D outperforms the prior arts on several benchmark datasets.

Figures (7)

• Figure 1: (a) Performance (RMSE error) curves on M3D test dataset Chang2017Matterport3DLF. Larger circles mean more parameters (e.g., ResNet18, ResNet34, ResNet50) and lower errors mean better performance. (b) Comparison with the ResNet34 as the ERP encoder backbone. With only 1M more parameters, our depth result is more accurate.
• Figure 2: Different projections of a spherical imaging panorama.
• Figure 3: (a) An overview of our Elite360D framework, comprising image-based ERP feature extraction (Sec. \ref{['sec:erp_feature']}), point-based ICOSAP feature extraction (Sec. \ref{['sec:ico_feature']}), and Bi-projection Bi-attention fusion (B2F) (Sec. \ref{['sec:b2f']}). For better visualization, we do not show the skip connections Ronneberger2015UNetCN at the decoding stage. (b) Illustration of the B2F module, consisting of three parts: semantic-aware affinity attention block (Fig. \ref{['fig:SAattenion']}), distance-aware affinity attention block (Fig. \ref{['fig:DAattenion']}) and gated fusion (Eq. \ref{['eq:gate_fusion']}).
• Figure 4: The subdivision of icosahedron at different resolution $l$.
• Figure 5: The architecture of semantic-aware affinity attention. Especially, $\mathbf{Q}^{S}_{i,j} \in \mathbb{R}^{1 \times d}$, $\mathbf{K}^{S} \in \mathbb{R}^{N \times d}$, and $\mathbf{V}^{S} \in \mathbb{R}^{N \times d}$, where $d$ is the dimension and $N$ is the ICOSAP point number.