C2PD: Continuity-Constrained Pixelwise Deformation for Guided Depth Super-Resolution

TL;DR

The paper tackles guided depth super-resolution by addressing the discontinuity of treating depth as discrete pixel values. It introduces CAPO, a Continuity-constrained Asymmetrical Pixelwise Operation, and PCGD, a Pixelwise Cross Gradient Deformation, to model depth as a continuous, isovolumetric-like deformation guided by RGB information. Together, CAPO and PCGD enable gradient-driven, continuous-depth upsampling that preserves spatial continuity and edge structure, achieving state-of-the-art results across four benchmarks and demonstrating strong generalization to large-scale tasks. The approach offers robust performance against RGB texture variability and presents a novel direction for depth upsampling in real-world, large-scale scenarios.

Abstract

Guided depth super-resolution (GDSR) has demonstrated impressive performance across a wide range of domains, with numerous methods being proposed. However, existing methods often treat depth maps as images, where shading values are computed discretely, making them struggle to effectively restore the continuity inherent in the depth map. In this paper, we propose a novel approach that maximizes the utilization of spatial characteristics in depth, coupled with human abstract perception of real-world substance, by transforming the GDSR issue into deformation of a roughcast with ideal plasticity, which can be deformed by force like a continuous object. Specifically, we firstly designed a cross-modal operation, Continuity-constrained Asymmetrical Pixelwise Operation (CAPO), which can mimic the process of deforming an isovolumetrically flexible object through external forces. Utilizing CAPO as the fundamental component, we develop the Pixelwise Cross Gradient Deformation (PCGD), which is capable of emulating operations on ideal plastic objects (without volume constraint). Notably, our approach demonstrates state-of-the-art performance across four widely adopted benchmarks for GDSR, with significant advantages in large-scale tasks and generalizability.

Figures (10)

• Figure 1: Illustration of some issues of GDSR and our method to address them. (a) After convolution, each point on residual is only obtained by corresponding area of input, which means the relationship of points in output is discrete. Residual connections also increase the difficulty of residual fitting. Those operations treat depth discretely, easy to induce deviation after reconstruction. (b) Our model established a systematic architecture for continuity-constrained deformation with fully integrated operational modules.
• Figure 2: Network architecture of our method. The Isovolumetric Deformation initially uses CAPO to deform the depth to organize pertinent information. Then, depth undergoes further processing through PCGD in the form of gradients.
• Figure 3: (a) Considering the interaction among particles, which preferentially convey collectively as a cohesive viscous fluid, the variation at each position is influenced by the combined trends in surrounding regions. (b) The principle of CAPO(1x4) with respect to local information. (c) Controlled with PCGD, each point only needs to focus on its own gradient changes, while the 2D macroscopic effect is regulated by guidance information with large contexts.
• Figure 4: To transfer upsampled LR (LR_up) to GT, we should move the values to right and shrink the transition area. These changes may be shown as the blue lines, where target values in (a) are all less or equal to the current values in (b), which means we cannot achieve GT by isovolumetric deformation. However, it can be completed by transferring gradient showed as orange line so that the values around position 3 in (a) are passed to position 5, with width shrinking.
• Figure 5: Qualitative comparison of x8 scale factor on four benchmark datasets.