Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$\mathbf{R}^3$: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain

Nate Rothschild, Moshe Kimhi, Avi Mendelson, Chaim Baskin

TL;DR

This work argues for deraining and low-level reconstruction to be performed in the raw Bayer domain to preserve sensor information before ISP distortions. It introduces a software ISP pipeline to enable fair comparisons with post-ISP RGB methods, and trains lightweight Bayer-domain deraining networks on a new Raw-Rain stereo dataset. A color-invariant Information Conservation Score (ICS), combining $MS$-$SSIM$ with a frequency-domain KL divergence between normalized power spectra, better captures perceptual and structural fidelity than traditional metrics. Results show Bayer-domain deraining can outperform RGB baselines (up to +0.99 dB PSNR and +1.2% ICS) with lower compute, supporting an ISP-last paradigm and opening avenues for end-to-end learnable camera pipelines and broader pre-ISP restoration tasks.

Abstract

Image reconstruction from corrupted images is crucial across many domains. Most reconstruction networks are trained on post-ISP sRGB images, even though the image-signal-processing pipeline irreversibly mixes colors, clips dynamic range, and blurs fine detail. This paper uses the rain degradation problem as a use case to show that these losses are avoidable, and demonstrates that learning directly on raw Bayer mosaics yields superior reconstructions. To substantiate the claim, we (i) evaluate post-ISP and Bayer reconstruction pipelines, (ii) curate Raw-Rain, the first public benchmark of real rainy scenes captured in both 12-bit Bayer and bit-depth-matched sRGB, and (iii) introduce Information Conservation Score (ICS), a color-invariant metric that aligns more closely with human opinion than PSNR or SSIM. On the test split, our raw-domain model improves sRGB results by up to +0.99 dB PSNR and +1.2% ICS, while running faster with half of the GFLOPs. The results advocate an ISP-last paradigm for low-level vision and open the door to end-to-end learnable camera pipelines.

$\mathbf{R}^3$: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain

TL;DR

This work argues for deraining and low-level reconstruction to be performed in the raw Bayer domain to preserve sensor information before ISP distortions. It introduces a software ISP pipeline to enable fair comparisons with post-ISP RGB methods, and trains lightweight Bayer-domain deraining networks on a new Raw-Rain stereo dataset. A color-invariant Information Conservation Score (ICS), combining - with a frequency-domain KL divergence between normalized power spectra, better captures perceptual and structural fidelity than traditional metrics. Results show Bayer-domain deraining can outperform RGB baselines (up to +0.99 dB PSNR and +1.2% ICS) with lower compute, supporting an ISP-last paradigm and opening avenues for end-to-end learnable camera pipelines and broader pre-ISP restoration tasks.

Abstract

Image reconstruction from corrupted images is crucial across many domains. Most reconstruction networks are trained on post-ISP sRGB images, even though the image-signal-processing pipeline irreversibly mixes colors, clips dynamic range, and blurs fine detail. This paper uses the rain degradation problem as a use case to show that these losses are avoidable, and demonstrates that learning directly on raw Bayer mosaics yields superior reconstructions. To substantiate the claim, we (i) evaluate post-ISP and Bayer reconstruction pipelines, (ii) curate Raw-Rain, the first public benchmark of real rainy scenes captured in both 12-bit Bayer and bit-depth-matched sRGB, and (iii) introduce Information Conservation Score (ICS), a color-invariant metric that aligns more closely with human opinion than PSNR or SSIM. On the test split, our raw-domain model improves sRGB results by up to +0.99 dB PSNR and +1.2% ICS, while running faster with half of the GFLOPs. The results advocate an ISP-last paradigm for low-level vision and open the door to end-to-end learnable camera pipelines.

Paper Structure

This paper contains 15 sections, 4 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Raw/Bayer-Domain Image Processing
  4. Single-Image Deraining
  5. Multi-view and Stereo Deraining & Reconstruction
  6. Methodology
  7. Software ISP Pipeline
  8. Deraining Model Architecture
  9. Color-Invariant Reconstruction Metric
  10. Spectral KL as an information–loss proxy.
  11. Dataset Collection and Composition
  12. Results
  13. Experimental setting
  14. Quantitative Comparison
  15. Conclusion

Figures (6)

  • Figure 1: Columns Left-to-Right: Original (GT) image, Corrupted (rain) image, Bayer pipeline reconstruction, sRGB pipeline reconstruction
  • Figure 2: Identical Pipeline Architectures, only deraining model location varies
  • Figure 3: Clockwise, Top-Left; Ground Truth, Rainy - raindrops skew WB/CCM statistics, RGB Pipeline - rain removal done after WB/CCM statistics, color similar to rainy image, Bayer Pipeline - Rain removal prior ISP, WB/CCM statistics similar to GT
  • Figure 4: Diverse Training Data. Metrics calculated for rainy images compared with GT
  • Figure 5: Identity Data real-world driving scenes, where input equal the gt. Allows for Synthetic data creation and training models for the identity operator
  • ...and 1 more figures