Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometry-Aware Sparse Depth Sampling for High-Fidelity RGB-D Depth Completion in Robotic Systems

Tony Salloom, Dandi Zhou, Xinhai Sun

TL;DR

The paper tackles the mismatch between realistic sensor sparsity and training-time depth completion by introducing a geometry-aware sparse depth sampling method. It uses PCA-based surface normals to compute a per-pixel depth reliability map and samples sparse depth according to this reliability, replacing uniform random sampling in a diffusion-based depth completion framework (Marigold-DC). Evaluated on NYU Depth v2, the approach yields meaningful RMSE/MAE improvements and clearer edge preservation, especially at mid-range sparse densities, suggesting more faithful sensor-inspired inputs for robotic perception. The work enhances robustness of 3D perception in industrial robotics and points to future integration of semantic priors and uncertainty to further tighten real-world applicability.

Abstract

Accurate three-dimensional perception is essential for modern industrial robotic systems that perform manipulation, inspection, and navigation tasks. RGB-D and stereo vision sensors are widely used for this purpose, but the depth maps they produce are often noisy, incomplete, or biased due to sensor limitations and environmental conditions. Depth completion methods aim to generate dense, reliable depth maps from RGB images and sparse depth input. However, a key limitation in current depth completion pipelines is the unrealistic generation of sparse depth: sparse pixels are typically selected uniformly at random from dense ground-truth depth, ignoring the fact that real sensors exhibit geometry-dependent and spatially nonuniform reliability. In this work, we propose a normal-guided sparse depth sampling strategy that leverages PCA-based surface normal estimation on the RGB-D point cloud to compute a per-pixel depth reliability measure. The sparse depth samples are then drawn according to this reliability distribution. We integrate this sampling method with the Marigold-DC diffusion-based depth completion model and evaluate it on NYU Depth v2 using the standard metrics. Experiments show that our geometry-aware sparse depth improves accuracy, reduces artifacts near edges and discontinuities, and produces more realistic training conditions that better reflect real sensor behavior.

Geometry-Aware Sparse Depth Sampling for High-Fidelity RGB-D Depth Completion in Robotic Systems

TL;DR

The paper tackles the mismatch between realistic sensor sparsity and training-time depth completion by introducing a geometry-aware sparse depth sampling method. It uses PCA-based surface normals to compute a per-pixel depth reliability map and samples sparse depth according to this reliability, replacing uniform random sampling in a diffusion-based depth completion framework (Marigold-DC). Evaluated on NYU Depth v2, the approach yields meaningful RMSE/MAE improvements and clearer edge preservation, especially at mid-range sparse densities, suggesting more faithful sensor-inspired inputs for robotic perception. The work enhances robustness of 3D perception in industrial robotics and points to future integration of semantic priors and uncertainty to further tighten real-world applicability.

Abstract

Accurate three-dimensional perception is essential for modern industrial robotic systems that perform manipulation, inspection, and navigation tasks. RGB-D and stereo vision sensors are widely used for this purpose, but the depth maps they produce are often noisy, incomplete, or biased due to sensor limitations and environmental conditions. Depth completion methods aim to generate dense, reliable depth maps from RGB images and sparse depth input. However, a key limitation in current depth completion pipelines is the unrealistic generation of sparse depth: sparse pixels are typically selected uniformly at random from dense ground-truth depth, ignoring the fact that real sensors exhibit geometry-dependent and spatially nonuniform reliability. In this work, we propose a normal-guided sparse depth sampling strategy that leverages PCA-based surface normal estimation on the RGB-D point cloud to compute a per-pixel depth reliability measure. The sparse depth samples are then drawn according to this reliability distribution. We integrate this sampling method with the Marigold-DC diffusion-based depth completion model and evaluate it on NYU Depth v2 using the standard metrics. Experiments show that our geometry-aware sparse depth improves accuracy, reduces artifacts near edges and discontinuities, and produces more realistic training conditions that better reflect real sensor behavior.

Paper Structure

This paper contains 21 sections, 13 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Depth Sensing in Robotics
  4. Depth Completion and Sparse-to-Dense Methods
  5. Limitations of Random Sparse Depth Sampling
  6. Research methodology
  7. Sources of Error in RGB-D and Stereo Depth Estimation
  8. Proposed Method: Normal-Based Sparse Depth Selection
  9. Point Cloud Extraction
  10. Surface Normal Estimation via PCA
  11. Depth Reliability Metric and Sampling Strategy
  12. Integration with the Marigold-DC Model
  13. Experimental Setup
  14. Dataset
  15. Integration with the Marigold-DC Model
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Applying a geometry-aware sparse depth selection method. The brighter the pixel, the higher its reliability.
  • Figure 2: Sparse depth creation algorithm flowchart.
  • Figure 3: The imapact of different sparse depth map size on depth completion results when using our geometry-aware depth selection and the uniform random selection method.