Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learned Lightweight Smartphone ISP with Unpaired Data

Andrei Arhire, Radu Timofte

TL;DR

This work tackles the challenge of learning a smartphone image signal processor (ISP) without pixel-perfect paired data by proposing an unpaired training framework. It combines content, color, and texture losses with three discriminators employing relativistic adversarial objectives to preserve structure while improving color and texture fidelity, using lightweight backbones for on-device inference. On Zurich RAW-to-RGB and Fujifilm UltraISP datasets, the unpaired method achieves high fidelity and competitive perceptual quality (LPIPS) compared to paired training, demonstrating practical viability for mobile cameras. The approach offers a path toward robust, data-efficient learned ISPs with potential enhancements from NILUT preprocessing and adaptive training strategies.

Abstract

The Image Signal Processor (ISP) is a fundamental component in modern smartphone cameras responsible for conversion of RAW sensor image data to RGB images with a strong focus on perceptual quality. Recent work highlights the potential of deep learning approaches and their ability to capture details with a quality increasingly close to that of professional cameras. A difficult and costly step when developing a learned ISP is the acquisition of pixel-wise aligned paired data that maps the raw captured by a smartphone camera sensor to high-quality reference images. In this work, we address this challenge by proposing a novel training method for a learnable ISP that eliminates the need for direct correspondences between raw images and ground-truth data with matching content. Our unpaired approach employs a multi-term loss function guided by adversarial training with multiple discriminators processing feature maps from pre-trained networks to maintain content structure while learning color and texture characteristics from the target RGB dataset. Using lightweight neural network architectures suitable for mobile devices as backbones, we evaluated our method on the Zurich RAW to RGB and Fujifilm UltraISP datasets. Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity across multiple evaluation metrics. The code and pre-trained models are available at https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data .

Learned Lightweight Smartphone ISP with Unpaired Data

TL;DR

This work tackles the challenge of learning a smartphone image signal processor (ISP) without pixel-perfect paired data by proposing an unpaired training framework. It combines content, color, and texture losses with three discriminators employing relativistic adversarial objectives to preserve structure while improving color and texture fidelity, using lightweight backbones for on-device inference. On Zurich RAW-to-RGB and Fujifilm UltraISP datasets, the unpaired method achieves high fidelity and competitive perceptual quality (LPIPS) compared to paired training, demonstrating practical viability for mobile cameras. The approach offers a path toward robust, data-efficient learned ISPs with potential enhancements from NILUT preprocessing and adaptive training strategies.

Abstract

The Image Signal Processor (ISP) is a fundamental component in modern smartphone cameras responsible for conversion of RAW sensor image data to RGB images with a strong focus on perceptual quality. Recent work highlights the potential of deep learning approaches and their ability to capture details with a quality increasingly close to that of professional cameras. A difficult and costly step when developing a learned ISP is the acquisition of pixel-wise aligned paired data that maps the raw captured by a smartphone camera sensor to high-quality reference images. In this work, we address this challenge by proposing a novel training method for a learnable ISP that eliminates the need for direct correspondences between raw images and ground-truth data with matching content. Our unpaired approach employs a multi-term loss function guided by adversarial training with multiple discriminators processing feature maps from pre-trained networks to maintain content structure while learning color and texture characteristics from the target RGB dataset. Using lightweight neural network architectures suitable for mobile devices as backbones, we evaluated our method on the Zurich RAW to RGB and Fujifilm UltraISP datasets. Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity across multiple evaluation metrics. The code and pre-trained models are available at https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data .
Paper Structure (16 sections, 7 equations, 5 figures, 4 tables)

This paper contains 16 sections, 7 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Content Loss
  5. Paired Color Loss
  6. Paired Texture Loss
  7. Relativistic Adversarial Color Loss
  8. Relativistic Adversarial Texture Loss
  9. Total Variation Loss
  10. Total Loss Function
  11. Experiments
  12. Dataset
  13. Ablation Study
  14. Backbones
  15. Implementation Details
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Overview of the winning architectures from the MAI 2021 and 2022 challenges. The dh_isp team ignatov2021learnedsmartphoneispmobile uses channel sizes [16, 16, 16], while MiAlgo ignatov2022learnedsmartphoneispmobile uses [12, 12, 12].
  • Figure 2: Overview of our proposed unpaired training method.
  • Figure 3: Architecture of the discriminators presented in \ref{['fig:architecture']}.
  • Figure 4: Dataset challenges. The first row shows images from the ZRR dataset (training subset) ignatov2020replacingmobilecameraisp, which include dynamic elements and slight viewpoint misalignments. The second row shows a Fujifilm UltraISP ignatov2022microispprocessing32mpphotos training sample with noticeable warping caused by the alignment algorithm.
  • Figure 5: Visual comparisons of outputs and target images on ZRR dataset (test subset) ignatov2020replacingmobilecameraisp. Last three columns show visual results of Efficient ISP trained under different data access settings.