Revisiting 360 Depth Estimation with PanoGabor: A New Fusion Perspective

TL;DR

This work tackles monocular 360-degree depth estimation by addressing distortions that arise when fusing panorama representations into ERP. It introduces PGFuse, which combines Distortion-aware PanoGabor Filters with a Channel-wise and Spatial-wise Unidirectional Fusion Module (CS-UFM) and leverages a latitude-based distortion model $K(\phi) = \frac{1}{\cos(\phi)} - 1$ to generate panorama-specific filters. A spherical gradient constraint stabilizes the orientation sensitivity of the Gabor filters, while a dual-cubemap input (original and 45° rotated) is fused into ERP via CS-UFM and MS-DAM to produce distortion-free depth maps. Across three indoor datasets, PGFuse achieves state-of-the-art performance with favorable efficiency, demonstrating robust depth estimation under panoramic distortion and large FoV.

Abstract

Depth estimation from a monocular 360 image is important to the perception of the entire 3D environment. However, the inherent distortion and large field of view (FoV) in 360 images pose great challenges for this task. To this end, existing mainstream solutions typically introduce additional perspective-based 360 representations (\textit{e.g.}, Cubemap) to achieve effective feature extraction. Nevertheless, regardless of the introduced representations, they eventually need to be unified into the equirectangular projection (ERP) format for the subsequent depth estimation, which inevitably reintroduces the troublesome distortions. In this work, we propose an oriented distortion-aware Gabor Fusion framework (PGFuse) to address the above challenges. First, we introduce Gabor filters that analyze texture in the frequency domain, thereby extending the receptive fields and enhancing depth cues. To address the reintroduced distortions, we design a linear latitude-aware distortion representation method to generate customized, distortion-aware Gabor filters (PanoGabor filters). Furthermore, we design a channel-wise and spatial-wise unidirectional fusion module (CS-UFM) that integrates the proposed PanoGabor filters to unify other representations into the ERP format, delivering effective and distortion-free features. Considering the orientation sensitivity of the Gabor transform, we introduce a spherical gradient constraint to stabilize this sensitivity. Experimental results on three popular indoor 360 benchmarks demonstrate the superiority of the proposed PGFuse to existing state-of-the-art solutions. Code can be available upon acceptance.

Figures (10)

• Figure 1: Comparison of our proposed framework (shown in red) against other existing methods. Our method consistently demonstrates lower values across all error metrics. The compact red area highlights that our approach maintains competitive scores in computational efficiency (FLOPs), demonstrating a well-balanced trade-off between performance and computational cost.
• Figure 2: Overview of the proposed framework. Our framework takes a single panorama as input and outputs the corresponding depth map. This network addresses the reintroduced distortion problems that are often overlooked by previous fusion-based methods. Specifically, we introduce a PanoGabor approach to achieve the distortion awareness.
• Figure 3: Illustration of the differential area element on the spherical surface (left) and the ERP domain (right).
• Figure 4: Visualization of the features extracted through the vanilla CNN layer and our proposed PanoGabor layer. Compared to the vanilla CNN layer, our PanoGabor layer is more robust regarding surface textures and complex illumination variations, which is crucial for accurate indoor depth estimation.
• Figure 5: Illustration of the spherical tangent sampling. a represent that the pixels are projected from the pixel coordinates to the spherical coordinates to sampling tangent points. b indicates that the calculated sampling points are projected back to the pixel coordinates.