Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D Scene Geometry Estimation from 360$^\circ$ Imagery: A Survey

Thiago Lopes Trugillo da Silveira, Paulo Gamarra Lessa Pinto, Jeffri Erwin Murrugarra Llerena, Claudio Rosito Jung

TL;DR

The paper surveys 3D scene geometry estimation from omnidirectional 360° imagery, detailing foundations of the spherical camera model and common representations, and then reviews monocular, stereo, and multi-view approaches on panoramas. It highlights the dominance of learning-based methods for single-view depth and layout estimation, surveys a wide range of representation schemes (ERP, CMP, tangent planes, icospheres, SpherePHD), and discusses datasets and evaluation metrics. Key contributions include a structured taxonomy of panorama-based 3D reconstruction methods, a consolidated view of public datasets, and a critical assessment of state-of-the-art results across indoor and outdoor scenarios. The survey underscores the need for standardized benchmarks and scalable spherical learning approaches to enable robust, real-time 6-DoF immersive navigation in AR/VR/MR and autonomous systems.

Abstract

This paper provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies based on single, two, or multiple images captured under the omnidirectional optics. We first revisit the basic concepts of the spherical camera model, and review the most common acquisition technologies and representation formats suitable for omnidirectional (also called 360$^\circ$, spherical or panoramic) images and videos. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data. The classical stereo matching is then revised on the spherical domain, where methodologies for detecting and describing sparse and dense features become crucial. The stereo matching concepts are then extrapolated for multiple view camera setups, categorizing them among light fields, multi-view stereo, and structure from motion (or visual simultaneous localization and mapping). We also compile and discuss commonly adopted datasets and figures of merit indicated for each purpose and list recent results for completeness. We conclude this paper by pointing out current and future trends.

3D Scene Geometry Estimation from 360$^\circ$ Imagery: A Survey

TL;DR

The paper surveys 3D scene geometry estimation from omnidirectional 360° imagery, detailing foundations of the spherical camera model and common representations, and then reviews monocular, stereo, and multi-view approaches on panoramas. It highlights the dominance of learning-based methods for single-view depth and layout estimation, surveys a wide range of representation schemes (ERP, CMP, tangent planes, icospheres, SpherePHD), and discusses datasets and evaluation metrics. Key contributions include a structured taxonomy of panorama-based 3D reconstruction methods, a consolidated view of public datasets, and a critical assessment of state-of-the-art results across indoor and outdoor scenarios. The survey underscores the need for standardized benchmarks and scalable spherical learning approaches to enable robust, real-time 6-DoF immersive navigation in AR/VR/MR and autonomous systems.

Abstract

This paper provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies based on single, two, or multiple images captured under the omnidirectional optics. We first revisit the basic concepts of the spherical camera model, and review the most common acquisition technologies and representation formats suitable for omnidirectional (also called 360, spherical or panoramic) images and videos. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data. The classical stereo matching is then revised on the spherical domain, where methodologies for detecting and describing sparse and dense features become crucial. The stereo matching concepts are then extrapolated for multiple view camera setups, categorizing them among light fields, multi-view stereo, and structure from motion (or visual simultaneous localization and mapping). We also compile and discuss commonly adopted datasets and figures of merit indicated for each purpose and list recent results for completeness. We conclude this paper by pointing out current and future trends.
Paper Structure (27 sections, 3 equations, 12 figures, 6 tables)

This paper contains 27 sections, 3 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Comparison with Existing Surveys
  3. Organization of this Paper
  4. Foundations, Acquisition and Representation of Spherical Images
  5. Spherical imaging
  6. Acquisition and Representation
  7. Monocular 3D Geometry Estimation
  8. Spherical Neural Networks
  9. Layout Estimation
  10. Depth Estimation
  11. Stereoscopic 3D Geometry Estimation
  12. Sparse Feature Extraction and Matching
  13. Dense Feature Extraction and Matching
  14. Depth Estimation
  15. Multi-view 3D Geometry Estimation
  16. ...and 12 more sections

Figures (12)

  • Figure 1: Paper Organization Diagram
  • Figure 2: Projection of a world point onto two spherical cameras with different extrinsic parameters.
  • Figure 3: Popular omnidirectional capture setups. Dotted cylindroids represent the FoV of each lens.
  • Figure 4: Ominidirectional image representations: (a) a spherical image (projected orthographically) and its mapping to (b) equirectangular and (c) cube-map formats. Sample image taken from the SUN360 dataset Xiao:CVPR:2012.
  • Figure 5: Examples of layout estimation results. Color image (personal dataset) and different layout representations are overlaid for better visualization using (a) DuLa-NetYang:CVPR:2019, (b) HorizonNetSun:CVPR:2019, (c) Corners for LayoutFernandez-Labrador:LRA:2020, and (d) HoHoNetsun:arxiv:2020. Layout markers are depicted as meant by the authors in their publicly available codes.
  • ...and 7 more figures