RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods

TL;DR

This survey addresses RGB guided ToF imaging, focusing on how RGB content guides depth enhancement in two tasks: guided depth super-resolution (GDSR) and guided depth completion (GDC). It systematically analyzes network architectures, learning paradigms, loss functions, and evaluation metrics, and provides quantitative comparisons on standard benchmarks. The authors identify major trends (dual-branch designs, multi-scale and auxiliary guidance, and fusion strategies) while highlighting challenges in unsupervised learning, cross-domain generalization, and real-world deployment. Practical directions include lightweight offline/online models, robust RGB-depth alignment, and multi-frame processing, all aimed at making RGB guided ToF systems more accurate, efficient, and deployable in diverse applications.

Abstract

Integrating an RGB camera into a ToF imaging system has become a significant technique for perceiving the real world. The RGB guided ToF imaging system is crucial to several applications, including face anti-spoofing, saliency detection, and trajectory prediction. Depending on the distance of the working range, the implementation schemes of the RGB guided ToF imaging systems are different. Specifically, ToF sensors with a uniform field of illumination, which can output dense depth but have low resolution, are typically used for close-range measurements. In contrast, LiDARs, which emit laser pulses and can only capture sparse depth, are usually employed for long-range detection. In the two cases, depth quality improvement for RGB guided ToF imaging corresponds to two sub-tasks: guided depth super-resolution and guided depth completion. In light of the recent significant boost to the field provided by deep learning, this paper comprehensively reviews the works related to RGB guided ToF imaging, including network structures, learning strategies, evaluation metrics, benchmark datasets, and objective functions. Besides, we present quantitative comparisons of state-of-the-art methods on widely used benchmark datasets. Finally, we discuss future trends and the challenges in real applications for further research.

Figures (15)

• Figure 1: An example of LiDAR point cloud from the KITTI dataset reprojected on images at different resolutions.
• Figure 2: Timeline showing the advances in RGB guided ToF Imaging. Methods are marked in blue or purple, depending on the learning paradigm being unsupervised or supervised.
• Figure 3: iToF and dToF working principles. The former computes distances based on the phase shift between emitted and reflected light; the latter measures the time required for a laser light to bounce over an object.
• Figure 4: Schematic diagram of LiDAR systems with different imaging principles.
• Figure 5: Taxonomy of RGB guided ToF imaging techniques. We identify 5 and 6 main categories for GDSR and GDC methods.