Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perspective-aware fusion of incomplete depth maps and surface normals for accurate 3D reconstruction

Ondrej Hlinka, Georg Kaniak, Christian Kapeller

TL;DR

The paper tackles accurate 3D surface reconstruction from depth and surface normals captured by a single-perspective camera, where perspective projection distorts conventional orthographic fusion. It introduces a perspective-aware log-depth fusion framework, enabling the reuse of gradient-based orthographic solvers to jointly fuse depth data and normals, while naturally handling missing depth via normals-based inpainting. The approach includes a TGV-regularized variant that further suppresses noise and staircasing, with the key step being a log-depth substitution $l(u,v)=\ln d(u,v)$ and a perspective gradient relation derived from normals. Evaluation on the DiLiGenT-MV dataset demonstrates that perspective-aware methods, particularly PTGV, outperform orthographic baselines and naive approaches, highlighting the practical importance of incorporating perspective effects in depth–normals fusion. The resulting method yields metrically accurate reconstructions suitable for real-time single-view 3D sensing tasks where depth gaps occur due to occlusions or material properties.

Abstract

We address the problem of reconstructing 3D surfaces from depth and surface normal maps acquired by a sensor system based on a single perspective camera. Depth and normal maps can be obtained through techniques such as structured-light scanning and photometric stereo, respectively. We propose a perspective-aware log-depth fusion approach that extends existing orthographic gradient-based depth-normals fusion methods by explicitly accounting for perspective projection, leading to metrically accurate 3D reconstructions. Additionally, the method handles missing depth measurements by leveraging available surface normal information to inpaint gaps. Experiments on the DiLiGenT-MV data set demonstrate the effectiveness of our approach and highlight the importance of perspective-aware depth-normals fusion.

Perspective-aware fusion of incomplete depth maps and surface normals for accurate 3D reconstruction

TL;DR

The paper tackles accurate 3D surface reconstruction from depth and surface normals captured by a single-perspective camera, where perspective projection distorts conventional orthographic fusion. It introduces a perspective-aware log-depth fusion framework, enabling the reuse of gradient-based orthographic solvers to jointly fuse depth data and normals, while naturally handling missing depth via normals-based inpainting. The approach includes a TGV-regularized variant that further suppresses noise and staircasing, with the key step being a log-depth substitution and a perspective gradient relation derived from normals. Evaluation on the DiLiGenT-MV dataset demonstrates that perspective-aware methods, particularly PTGV, outperform orthographic baselines and naive approaches, highlighting the practical importance of incorporating perspective effects in depth–normals fusion. The resulting method yields metrically accurate reconstructions suitable for real-time single-view 3D sensing tasks where depth gaps occur due to occlusions or material properties.

Abstract

We address the problem of reconstructing 3D surfaces from depth and surface normal maps acquired by a sensor system based on a single perspective camera. Depth and normal maps can be obtained through techniques such as structured-light scanning and photometric stereo, respectively. We propose a perspective-aware log-depth fusion approach that extends existing orthographic gradient-based depth-normals fusion methods by explicitly accounting for perspective projection, leading to metrically accurate 3D reconstructions. Additionally, the method handles missing depth measurements by leveraging available surface normal information to inpaint gaps. Experiments on the DiLiGenT-MV data set demonstrate the effectiveness of our approach and highlight the importance of perspective-aware depth-normals fusion.
Paper Structure (9 sections, 17 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 9 sections, 17 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Naive perspective fusion
  3. Related work
  4. Contribution
  5. Definitions and notation
  6. Orthographic fusion
  7. Proposed perspective fusion
  8. Performance evaluation
  9. Conclusion

Figures (1)

  • Figure 1: (a) Ground truth depth data for the reading object from the DiLiGenT-MV data set. (b) Lower-resolution depth data perturbed with noise and missing depth information. (c) Confidence map $\kappa$. (d) RGB color-coded surface normal map perturbed with noise. (e–h) Surface reconstructions using Ortho (e), Naive (f), PG (g), and PTGV (h) fusion methods applied to (b) and (d).