Point Cloud Color Constancy

TL;DR

Point Cloud Color Constancy (PCCC) reframes illumination estimation as a 3D regression task on RGB-D point clouds derived from synchronized RGB and ToF data. Using a PointNet-based regressor with a spatial weighting mechanism, PCCC predicts per-point illuminants and aggregates them into a global illumination vector, achieving state-of-the-art accuracy while running at over 500 fps on thumbnail inputs. The approach is validated on relabeled RGB-D benchmarks (NYU-v2, DIODE, ETH3D) and a new DepthAWB dataset, demonstrating substantial improvements over prior single-image and depth-assisted methods. By exploiting depth geometry, PCCC enables robust, hardware-friendly color constancy with potential extensions to local AWB via per-point illumination maps, offering practical benefits for mobile cameras and ISP pipelines.

Abstract

In this paper, we present Point Cloud Color Constancy, in short PCCC, an illumination chromaticity estimation algorithm exploiting a point cloud. We leverage the depth information captured by the time-of-flight (ToF) sensor mounted rigidly with the RGB sensor, and form a 6D cloud where each point contains the coordinates and RGB intensities, noted as (x,y,z,r,g,b). PCCC applies the PointNet architecture to the color constancy problem, deriving the illumination vector point-wise and then making a global decision about the global illumination chromaticity. On two popular RGB-D datasets, which we extend with illumination information, as well as on a novel benchmark, PCCC obtains lower error than the state-of-the-art algorithms. Our method is simple and fast, requiring merely 16*16-size input and reaching speed over 500 fps, including the cost of building the point cloud and net inference.

Figures (14)

• Figure 1: We present Point Cloud Color Constancy (PCCC), which is simple, efficient and hardware friendly. (A): Raw RGB-D image. (B): RGB point cloud generated by (A). (C): Point Cloud with illumination rendered from PCCC. (D) 2D illuminant map extracted from (C). (E) Color corrected image. (F): Speed, accuracy and parameter number comparison with state-of-the-art color constancy methods.
• Figure 2: The PCCC architecture consists of a point-wise feature extraction block - a slight modification of PointNetqi2017pointnet, and the spatial weighted illuminant estimation block. Given $H \times W$ points as input, PCCC outputs a spatial weighted illuminant, which benefits global auto white balance, AWB (Section \ref{['Sec:QR']}). With a slight setting change, we can also achieve point-wise illumination estimation (Section \ref{['Sec:Local']}).
• Figure 3: We use two augmentation approaches: (A) camera pose augmentation simulates images capture from a different viewing angle. (B) light intensity augmentation generates new pixel-wise illumination field under the same illuminant.
• Figure 4: (A)-(C): Illumination distribution of NUS-600D dataset, our relabel ETH3D dataset and propose dataset. (D), (E) a pair of images we captured sequentially. We use (D) with color checker for labelling and (E) for training and testing. (F) ToF depth map.
• Figure 5: Illustration of how the depth affects the final decision. Depth information helps PCCC to estimate illumination from more meaningful information (building wall) instead of noisy information (bright sky with disturbing color).