Joint RGB-Spectral Decomposition Model Guided Image Enhancement in Mobile Photography

TL;DR

The paper tackles the challenge of leveraging spectral information from mobile sensors to improve tone enhancement in outdoor HDR scenes. It introduces a joint RGB-Spectral decomposition model to extract shading $S$, reflectance $R$, and material priors $M$ from RGB and low-resolution spectral imagery, then guides HDRNet-based enhancement with these priors. The Mobile-Spec dataset provides aligned RGB-hyperspectral data, shading/reflectance maps, and material segmentation to support learning. The proposed JDM-HDRNet, featuring a localized shading module, Spectral Perception Self-Attention, and a Mixture of Semantic Grid Experts, outperforms prior methods in PSNR, SSIM, and color accuracy, and demonstrates improved generalization on unseen scenes. This work establishes a foundation for incorporating spectral information into mobile photography, with practical impact on real-time tone mapping and material-aware color rendering.

Abstract

The integration of miniaturized spectrometers into mobile devices offers new avenues for image quality enhancement and facilitates novel downstream tasks. However, the broader application of spectral sensors in mobile photography is hindered by the inherent complexity of spectral images and the constraints of spectral imaging capabilities. To overcome these challenges, we propose a joint RGB-Spectral decomposition model guided enhancement framework, which consists of two steps: joint decomposition and prior-guided enhancement. Firstly, we leverage the complementarity between RGB and Low-resolution Multi-Spectral Images (Lr-MSI) to predict shading, reflectance, and material semantic priors. Subsequently, these priors are seamlessly integrated into the established HDRNet to promote dynamic range enhancement, color mapping, and grid expert learning, respectively. Additionally, we construct a high-quality Mobile-Spec dataset to support our research, and our experiments validate the effectiveness of Lr-MSI in the tone enhancement task. This work aims to establish a solid foundation for advancing spectral vision in mobile photography. The code is available at \url{https://github.com/CalayZhou/JDM-HDRNet}.

Figures (16)

• Figure 1: The Lr-MSI is decomposed into the illumination $L(\lambda)$, reflectance $R(\lambda, x)$ and shading $S(x)$ terms. We then analyze their respective roles in the mobile ISP pipeline.
• Figure 1: More visualized samples of the Mobile-Spec dataset. The 16-bit RGB images are linear tone-mapped for visualization.
• Figure 2: (a) The Mobile-Spec dataset comprises RGB images (16-bit input and 8-bit target), hyperspectral images and their corresponding shading, reflectance and material segmentation images. Near-infrared images serve as the guide map to approximate the shading term. (b) Spectral responses of different material categories. (c-d) Visualizations of RGB and hyperspectral data for different categories with t-SNE van2008visualizing.
• Figure 2: (a) The dual camera system which consists of the high-end commercia smartphone and the GaiaSky-mini2 hyperspectral camera dualixGaiaskyminiHyperspectral. (b) Image Matching: the overlapping region is detected by the SIFT descriptor lowe2004distinctive, then the affine transformation is performed on the smartphone-captured RGB image to align with the pseudo-RGB image from the hyperspectral data. (c) The aligned pair of RGB and hyperspectral images, the hyperspectral image is transformed into the pseudo-RGB image.
• Figure 3: (a) The joint RGB-Spectral decomposition model leverages the complementarity between RGB images and Lr-MSIs to predict shading (S), reflectance (R) and material category (M) priors. (b) The pixel intensity histogram. (c-d) The distribution of Pearson correlation coefficient $\rho$ in the original RGB space and reflectance space.