Combining Pre- and Post-Demosaicking Noise Removal for RAW Video

TL;DR

This work addresses robust RAW video denoising across varying noise levels and scenes by introducing a two-stage pipeline that denoises both in the RAW mosaicked domain and after demosaicking, controlled by a tunable balance $ \alpha $. It combines self-similarity-based denoising with temporal trajectory prefiltering and a PCA-based spatio-temporal denoising block, guided by a sensor noise model to adapt to realistic Poisson-Gaussian noise. The approach demonstrates competitive performance against state-of-the-art deep learning methods on real and synthetic data while offering lower memory usage and greater adaptability, making it practical for real-world videography and potential mobile deployment. Overall, the method provides a robust, efficient framework for RAW video denoising that respects CFA structure and noise correlations while preserving fine textures.

Abstract

Denoising is one of the fundamental steps of the processing pipeline that converts data captured by a camera sensor into a display-ready image or video. It is generally performed early in the pipeline, usually before demosaicking, although studies swapping their order or even conducting them jointly have been proposed. With the advent of deep learning, the quality of denoising algorithms has steadily increased. Even so, modern neural networks still have a hard time adapting to new noise levels and scenes, which is indispensable for real-world applications. With those in mind, we propose a self-similarity-based denoising scheme that weights both a pre- and a post-demosaicking denoiser for Bayer-patterned CFA video data. We show that a balance between the two leads to better image quality, and we empirically find that higher noise levels benefit from a higher influence pre-demosaicking. We also integrate temporal trajectory prefiltering steps before each denoiser, which further improve texture reconstruction. The proposed method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level, and is competitive with the state of the art, making it suitable for real-world videography.

Figures (9)

• Figure 1: Architecture of the proposed denoising pipeline. We balance two denoising stages, one before and one after demosaicking, returning a portion $\alpha$ of the noise after the first one. Each stage performs, in a color decorrelated domain, a patch trajectory prefiltering and two iterations of a patch-based spatio-temporal denoising, only requiring an estimation of the sensor noise model.
• Figure 2: Diagram of the trajectory prefiltering sub-block. Patch trajectories are collected motion-warping adjacent frames and replacing occluded patches with neighbouring ones. The trajectories are filtered by thresholding in a weighted SVD domain, and they are all finally aggregated back into an image.
• Figure 3: Visual quality comparison with varying values of $\alpha$. No prefiltering nor iteration sub-blocks are used. Denoising purely before demosaicking ($\alpha=0$) leaves checkerboard artifacts, while doing so purely after ($\alpha=1$) leaves spatially correlated residual noise. A combination of both prevents both problems. Higher ISOs have a lower optimal $\alpha$ value. Scene from the CRVD outdoor dataset.
• Figure 4: Visual quality comparison with different combinations of prefiltering sub-blocks before and after demosaicking. Spatio-temporal denoising sub-blocks are used before and after demosaicking with $\alpha=0.5$. No iteration sub-blocks are used. Temporally prefiltering before and after demosaicking allows a better reconstruction of finer details. Scene from the CRVD outdoor dataset.
• Figure 5: Visual quality comparison with different combinations of iteration sub-blocks before and after demosaicking. Prefiltering and spatio-temporal denoising sub-blocks are used before and after demosaicking with $\alpha=0.5$. Iterating before and after demosaicking clean residual low frequency noise in flat areas (blue patch) without affecting textured areas (orange patch). Scene from the CRVD outdoor dataset.