Leveraging Multispectral Sensors for Color Correction in Mobile Cameras

TL;DR

This work tackles color correction for mobile cameras by fusing high-resolution RGB data with a low-resolution multispectral signal in an end-to-end framework. The authors adapt two lightweight image-to-image backbones (LPIENet and cmKAN) with a spectral encoder to jointly estimate illumination, discount it, and perform color space transformation, producing color-accurate CIE XYZ outputs. They introduce a physically grounded dataset of 116,688 RGB–MS–XYZ triplets under 102 illuminants, plus a misaligned variant, and demonstrate substantial improvements (up to ~50% in $\Delta E_{00}$) over RGB-only and MS baselines, including robustness to exposure changes. The results validate the value of incorporating multispectral cues throughout the color-correction pipeline and highlight the framework’s flexibility to accommodate future backbones and hardware configurations.

Abstract

Recent advances in snapshot multispectral (MS) imaging have enabled compact, low-cost spectral sensors for consumer and mobile devices. By capturing richer spectral information than conventional RGB sensors, these systems can enhance key imaging tasks, including color correction. However, most existing methods treat the color correction pipeline in separate stages, often discarding MS data early in the process. We propose a unified, learning-based framework that (i) performs end-to-end color correction and (ii) jointly leverages data from a high-resolution RGB sensor and an auxiliary low-resolution MS sensor. Our approach integrates the full pipeline within a single model, producing coherent and color-accurate outputs. We demonstrate the flexibility and generality of our framework by refactoring two different state-of-the-art image-to-image architectures. To support training and evaluation, we construct a dedicated dataset by aggregating and repurposing publicly available spectral datasets, rendering under multiple RGB camera sensitivities. Extensive experiments show that our approach improves color accuracy and stability, reducing error by up to 50% compared to RGB-only and MS-driven baselines. Datasets, code, and models will be made available upon acceptance.

Figures (4)

• Figure 1: Top: Overview of the proposed color correction framework. A high-resolution RGB sensor is paired with a low-resolution multispectral sensor within a camera module. The two inputs are fused by a unified model that jointly performs illuminant estimation, illuminant discounting, and color correction to produce a color-accurate output. Bottom: Visual comparison between our proposed approach and a classic correction pipeline (using FC$^4$hu2017fc4 for illuminant estimation). We report the average $\Delta$E$_{00}$ color distance and convert the images to sRGB for visualization purposes.
• Figure 2: Overview of the proposed architectures adapted from (a) LPIENet conde2023perceptual and (b) cmKAN nikonorov2025color. For both architectures, we add a spectral encoder module and fuse the features with the ones extracted from the RGB image. For visualization purposes the output and the input RGB images are respectively converted to sRGB and gamma corrected.
• Figure 3: (a) Overview of the dataset generation pipeline. Hyperspectral reflectance images are rendered using multiple camera sensitivities under different illuminants with known spectral power distributions (SPDs) to produce RGB, MS and GT image triplets. (b) Representative samples of the proposed dataset. For visualization purposes, images are converted to sRGB.
• Figure 4: Qualitative results of the best performing methods: FC$^{4}$hu2017fc4, SpectralConvMean Gong2019ConvolutionalMA, and our three proposed models. We show results from two mirrorless and two mobile cameras, with $\Delta$E$_{00}$ map reported in the bottom-right corner and average $\Delta$E$_{00}$ reported in the bottom-left one. For visualization purposes, we gamma correct the raw visualizations (first column) and convert the results (columns two to six) to sRGB. More qualitative results are reported in the Supplementary Material.