An End-to-End Real-World Camera Imaging Pipeline

TL;DR

This work deeply analyze coordinate-dependent optical distortions, e.g., vignetting and dark shading, and design a novel Coordinate-Aware Distortion Restoration (CADR) module to restore coordinate-biased distortions, and proposes a Coordinate-Independent Mapping Compression (CIMC) module to implement tone mapping and redundant information compression.

Abstract

Recent advances in neural camera imaging pipelines have demonstrated notable progress. Nevertheless, the real-world imaging pipeline still faces challenges including the lack of joint optimization in system components, computational redundancies, and optical distortions such as lens shading.In light of this, we propose an end-to-end camera imaging pipeline (RealCamNet) to enhance real-world camera imaging performance. Our methodology diverges from conventional, fragmented multi-stage image signal processing towards end-to-end architecture. This architecture facilitates joint optimization across the full pipeline and the restoration of coordinate-biased distortions. RealCamNet is designed for high-quality conversion from RAW to RGB and compact image compression. Specifically, we deeply analyze coordinate-dependent optical distortions, e.g., vignetting and dark shading, and design a novel Coordinate-Aware Distortion Restoration (CADR) module to restore coordinate-biased distortions. Furthermore, we propose a Coordinate-Independent Mapping Compression (CIMC) module to implement tone mapping and redundant information compression. Existing datasets suffer from misalignment and overly idealized conditions, making them inadequate for training real-world imaging pipelines. Therefore, we collected a real-world imaging dataset. Experiment results show that RealCamNet achieves the best rate-distortion performance with lower inference latency.

Figures (7)

• Figure 1: Coordinate-dependent distortion. The left side is the captured image, and the right side is the vignetting distribution map. The middle of the picture on the left is brighter.
• Figure 2: Ours Framework. The encoder $\boldsymbol{E}$ of RealCamNet proposes $\boldsymbol{CADR}$ to restorate coordinate-related distortion and builds $\boldsymbol{CIMC}$ to complete coordinate-independent functions (such as global and local tone mapping, denoising, and feature compression). The decoder $\boldsymbol{D}$ of RealCamNet proposes $\boldsymbol{CSA}$ to decode the decoded features and restore the RGB image. $\boldsymbol{LFT}$ is local feature transform, and $\boldsymbol{GFT}$ is global feature transform.
• Figure 3: Compared with previous methods that can only encode the relative coordinates of the cropped image, our method calculates the absolute coordinates of the cropped RAW image. Therefore our method can recover fixed position-type distortion in the image.
• Figure 4: The detail of the Color Prior Encoder (CPE).
• Figure 5: (a) Quantitative Rate-Distortion curve results. (b) Inference time results. (c) Ablation results.