OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations

TL;DR

OGNI-DC introduces Optimization-Guided Neural Iterations to depth completion by coupling a ConvGRU-based gradient refinement with a Differentiable Depth Integrator, enabling end-to-end training and strong generalization across unseen sparsities and datasets. The method predicts depth gradients rather than direct depths, enforcing sparse-depth consistency through a differentiable 2D least-squares integration and refining results via up-sampling and SPN-based refinement. Extensive experiments show state-of-the-art in-domain performance on NYUv2 and KITTI, plus robust zero-shot generalization to VOID and DDAD and strong sparsity-robustness across multiple datasets and sparsity patterns. The combination of optimization-based inductive bias with neural flexibility yields a single model that performs well across diverse environments and depth densities, with detailed ablations validating the contributions of DDI, iterative refinement, and supervision strategies.

Abstract

Depth completion is the task of generating a dense depth map given an image and a sparse depth map as inputs. It has important applications in various downstream tasks. In this paper, we present OGNI-DC, a novel framework for depth completion. The key to our method is "Optimization-Guided Neural Iterations" (OGNI). It consists of a recurrent unit that refines a depth gradient field and a differentiable depth integrator that integrates the depth gradients into a depth map. OGNI-DC exhibits strong generalization, outperforming baselines by a large margin on unseen datasets and across various sparsity levels. Moreover, OGNI-DC has high accuracy, achieving state-of-the-art performance on the NYUv2 and the KITTI benchmarks. Code is available at https://github.com/princeton-vl/OGNI-DC.

Figures (10)

• Figure 1: The overall pipeline of OGNI-DC. We first extract features at $1/4$ resolution from the concatenation of the image and the sparse depth map. After that, a ConvGRU iteratively refines a depth gradient field based on current predictions, and the DDI (\ref{['sec:DDI']}) integrates the depth gradient field into an intermediate depth map. Finally, we up-sample the intermediate depth map and enhance it with a Spatial Propagation Network (SPN) lin2022dynamic to get the full-resolution depth map.
• Figure 2: We demonstrate the effectiveness of our iterative refinement. The depth and the gradients predictions in the highlighted areas gradually improved. Red means negative gradients and green means positive gradients. Brighter colors mean larger absolute values. Some ground truths are missing in the gradients map due to incomplete depth.
• Figure 3: Qualitative comparisons of zero-shot generalization on the VOID void and the DDAD ddad datasets. The sparse depth observations are superimposed on the RGB images. Compared to baselines, our results are less noisy and sharper at boundaries.
• Figure 4: Qualitative comparison with LRRU wang2023lrru and CFormer zhang2023completionformer on the KITTI kittidc test split. Our method reconstructs the telephone pole better than the baselines.
• Figure a: Our model learns a confidence map (h) to filter out the noisy sparse observations in (g). Compare (e) to (f), the model with confidence prediction is more accurate.