PanDA: Towards Panoramic Depth Anything with Unlabeled Panoramas and Mobius Spatial Augmentation

TL;DR

The paper addresses panoramic monocular depth estimation by evaluating Depth Anything Models (DAMs) on $180^{\circ}\times360^{\circ}$ panoramas, identifying sensitivity to spherical distortions and the impact of representation, viewpoint, and geometry. It introduces PanDA, a semi-supervised approach that fine-tunes a DAM via LoRA on synthetic panoramas with an equator-aware loss (EPNL) and trains a student on unlabeled panoramas using pseudo-labels and Möbius transformation-based spatial augmentation (MTSA) with dedicated SSL losses. Key contributions include a thorough panorama-specific analysis, the EPNL and MTSA mechanisms, and state-of-the-art zero-shot panoramic depth performance on Matterport3D and Stanford2D3D, outperforming data-specific methods. The work offers a practical panoramic depth foundation model with robust generalization across scenes and distortions, enabling applications in VR, 360-degree perception, and autonomous systems.

Abstract

Recently, Depth Anything Models (DAMs) - a type of depth foundation models - have demonstrated impressive zero-shot capabilities across diverse perspective images. Despite its success, it remains an open question regarding DAMs' performance on panorama images that enjoy a large field-of-view (180x360) but suffer from spherical distortions. To address this gap, we conduct an empirical analysis to evaluate the performance of DAMs on panoramic images and identify their limitations. For this, we undertake comprehensive experiments to assess the performance of DAMs from three key factors: panoramic representations, 360 camera positions for capturing scenarios, and spherical spatial transformations. This way, we reveal some key findings, e.g., DAMs are sensitive to spatial transformations. We then propose a semi-supervised learning (SSL) framework to learn a panoramic DAM, dubbed PanDA. Under the umbrella of SSL, PanDA first learns a teacher model by fine-tuning DAM through joint training on synthetic indoor and outdoor panoramic datasets. Then, a student model is trained using large-scale unlabeled data, leveraging pseudo-labels generated by the teacher model. To enhance PanDA's generalization capability, M"obius transformation-based spatial augmentation (MTSA) is proposed to impose consistency regularization between the predicted depth maps from the original and spatially transformed ones. This subtly improves the student model's robustness to various spatial transformations, even under severe distortions. Extensive experiments demonstrate that PanDA exhibits remarkable zero-shot capability across diverse scenes, and outperforms the data-specific panoramic depth estimation methods on two popular real-world benchmarks.

Figures (18)

• Figure 1: (a) Our PanDA exhibits impressive panoramic depth estimation results in open-world scenarios. The resolution of presented panoramas is 1008$\times$2016. (b)Top row: Spherical images with different zoom levels, and the corresponding depth predictions with perspective projection. Middle row: ERP image. Bottom row: Spherical images with different vertical rotation angles, and the corresponding depth predictions with perspective projection. Our PanDA is robust to spherical transformations and predicts fine-grained depths.
• Figure 2: Left: Cropped patch of a panorama with 30$^{\circ}$ vertical rotation. Top row: Depth predictions. Bottom row: Gradient maps of depth predictions to better illustrate the depth variances. PanDA predicts clearer depth boundaries of the car.
• Figure 3: Overview of the analysis of DAMs.
• Figure 4: Different panoramic representations and their predicted depths after projecting back to the ERP plane.
• Figure 5: We place the 360$^{\circ}$ camera at three different heights and locations. (1) Positioning on the ground. (2) Placing on the tripod. (3) Magnifying towards the desk. Results show that the occupancy of polar regions influences the depth estimation at the equator.