RGB-Phase Speckle: Cross-Scene Stereo 3D Reconstruction via Wrapped Pre-Normalization

TL;DR

The paper tackles cross-domain robustness in 3D reconstruction by introducing RGB-Phase Speckle, which embeds phase information into RGB speckle projections and employs phase pre-normalization to align inputs across scenes. It proposes a data-centric solution that combines a color-speckle projection pipeline with a normalization step, and evaluates performance on a newly collected RGB speckle dataset in addition to SceneFlow. Key contributions include the RGB color speckle encoding, the phase pre-normalization technique, and a large, challenging speckle dataset with sub-pixel ground truth, demonstrating improved cross-scene generalization for stereo matching networks. The work has practical implications for robust active-binocular 3D sensing in diverse environments, with potential hardware and deployment benefits for real-world measurement tasks.

Abstract

3D reconstruction garners increasing attention alongside the advancement of high-level image applications, where dense stereo matching (DSM) serves as a pivotal technique. Previous studies often rely on publicly available datasets for training, focusing on modifying network architectures or incorporating specialized modules to extract domain-invariant features and thus improve model robustness. In contrast, inspired by single-frame structured-light phase-shifting encoding, this study introduces RGB-Speckle, a cross-scene 3D reconstruction framework based on an active stereo camera system, designed to enhance robustness. Specifically, we propose a novel phase pre-normalization encoding-decoding method: first, we randomly perturb phase-shift maps and embed them into the three RGB channels to generate color speckle patterns; subsequently, the camera captures phase-encoded images modulated by objects as input to a stereo matching network. This technique effectively mitigates external interference and ensures consistent input data for RGB-Speckle, thereby bolstering cross-domain 3D reconstruction stability. To validate the proposed method, we conduct complex experiments: (1) construct a color speckle dataset for complex scenarios based on the proposed encoding scheme; (2) evaluate the impact of the phase pre-normalization encoding-decoding technique on 3D reconstruction accuracy; and (3) further investigate its robustness across diverse conditions. Experimental results demonstrate that the proposed RGB-Speckle model offers significant advantages in cross-domain and cross-scene 3D reconstruction tasks, enhancing model generalization and reinforcing robustness in challenging environments, thus providing a novel solution for robust 3D reconstruction research.

Figures (8)

• Figure 1: Comparison of the performance in challenging scenes. Columns from left to right denote sample input RGB image,the predicted disparities of the with rgb-speckle datasets,input phase image, the predicted disparities of the IGEV trained with phase pre-normalized images on the same datasets.
• Figure 2: The pipeline of our method. By projecting phase-encoded color speckle patterns onto the scene during image capture and applying phase pre-normalization, robust reconstruction can be achieved across different scenes using a stereo matching network trained on speckle dataset.
• Figure 3: The generation process of phase color speckle pattern is disturbed and upsampled in the same way on a phase-shifting fringe and merged into three RGB channels to realize the phase-shifting coding of a single image.
• Figure 4: In the scene of projected color speckle and the normalized image, it can be seen that the area projected speckle retains the phase information after pre-normalization, while the background area not projected speckle is ignored in this process.
• Figure 5: The production process of our speckle dataset. (a) A typical scene of our speckle dataset. (b)One of projected Gray code fringe. (c) Our projected color speckle. (d) The sub-pixel disparity ground truth.