Self-Distilled Depth Refinement with Noisy Poisson Fusion

TL;DR

The Self-distilled Depth Refinement (SDDR) framework is proposed to enforce robustness against the noises, which mainly consists of depth edge representation and edge-based guidance, and can acquire strong robustness to the noises.

Abstract

Depth refinement aims to infer high-resolution depth with fine-grained edges and details, refining low-resolution results of depth estimation models. The prevailing methods adopt tile-based manners by merging numerous patches, which lacks efficiency and produces inconsistency. Besides, prior arts suffer from fuzzy depth boundaries and limited generalizability. Analyzing the fundamental reasons for these limitations, we model depth refinement as a noisy Poisson fusion problem with local inconsistency and edge deformation noises. We propose the Self-distilled Depth Refinement (SDDR) framework to enforce robustness against the noises, which mainly consists of depth edge representation and edge-based guidance. With noisy depth predictions as input, SDDR generates low-noise depth edge representations as pseudo-labels by coarse-to-fine self-distillation. Edge-based guidance with edge-guided gradient loss and edge-based fusion loss serves as the optimization objective equivalent to Poisson fusion. When depth maps are better refined, the labels also become more noise-free. Our model can acquire strong robustness to the noises, achieving significant improvements in accuracy, edge quality, efficiency, and generalizability on five different benchmarks. Moreover, directly training another model with edge labels produced by SDDR brings improvements, suggesting that our method could help with training robust refinement models in future works.

Figures (16)

• Figure 1: (a) Visual comparisons. We model depth refinement by noisy Poisson fusion with the local inconsistency noise (representing the inconsistent billboard and wall in red box) and the edge deformation noise (indicating blurred depth edges in the blue box and second row). Better viewed when zoomed in. (b) Performance and efficiency. Circle area represents FLOPs. The two-stage methods boostdepthpatchfusion are reported by multiplying FLOPs per patch with patch numbers. SDDR outperforms prior arts in depth accuracy ($\delta_1$), edge quality (ORD), and model efficiency (FLOPs).
• Figure 2: Depiction of depth errors. We utilize two samples of high-quality depth maps as ideal depth $D^*$. For the predicted depth $D$, the combination of local inconsistency noise $\epsilon_{\text{cons}}$ and edge deformation noise $\epsilon_{\text{edge}}$ can approximate real depth error $D-D^*$ (the last two columns). Thus, as in the third and fourth columns, prediction $D$ can be depicted by the summation of $D^*$, $\epsilon_{\text{cons}}$, and $\epsilon_{\text{edge}}$.
• Figure 3: Overview of self-distilled depth refinement. SDDR consists of depth edge representation and edge-based guidance. Refinement network $\mathcal{N}_r$ produces initial refined depth $D_0$, edge representation $G_0$, and learnable soft mask $\Omega$ of high-frequency areas. The final depth edge representation $G_S$ is updated from coarse to fine as pseudo-labels. The edge-based guidance with edge-guided gradient loss and edge-based fusion loss supervises $\mathcal{N}_r$ to achieve consistent structures and fine-grained edges.
• Figure 3: Comparisons of model generalizability. We conduct zero-shot evaluations on DIML diml and DIODE diode datasets with diverse in-the-wild scenarios to compare the generalization capability. We adopt LeReS leres as the depth predictor for all the compared methods in this experiment.
• Figure 4: Visualization of intermediate results. We visualize the results of several important steps within the SDDR framework. The quantile sampling utilizes the same color map as in Fig. \ref{['fig:pipeline']}.