Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PQDynamicISP: Dynamically Controlled Image Signal Processor for Any Image Sensors Pursuing Perceptual Quality

Masakazu Yoshimura, Junji Otsuka, Takeshi Ohashi

TL;DR

PQDynamicISP introduces a sensor‑agnostic, perceptual‑quality ISP built from five lightweight conventional functions, a compact encoder, and a controller that dynamically sets ISP parameters. By combining global and local control mechanisms, it achieves per‑image and per‑region parameterization, enabling a universal ISP that works across unknown sensors. The training strategy includes automatic tuning of parameter search spaces and a denoiser‑as‑denoiser approach with a Local L1 loss to stabilize color fidelity, yielding robust performance. Empirical results on FiveK (universal ISP and enhancement), HDR+ (tone mapping), and MAI21 (normal ISP) demonstrate state‑of‑the‑art perceptual quality with substantially lower computational cost than large DNN ISPs, highlighting practical impact for edge devices and cloud pipelines and suggesting new directions for color‑mapping in DNNs.

Abstract

Full DNN-based image signal processors (ISPs) have been actively studied and have achieved superior image quality compared to conventional ISPs. In contrast to this trend, we propose a lightweight ISP that consists of simple conventional ISP functions but achieves high image quality by increasing expressiveness. Specifically, instead of tuning the parameters of the ISP, we propose to control them dynamically for each environment and even locally. As a result, state-of-the-art accuracy is achieved on various datasets, including other tasks like tone mapping and image enhancement, even though ours is lighter than DNN-based ISPs. Additionally, our method can process different image sensors with a single ISP through dynamic control, whereas conventional methods require training for each sensor.

PQDynamicISP: Dynamically Controlled Image Signal Processor for Any Image Sensors Pursuing Perceptual Quality

TL;DR

PQDynamicISP introduces a sensor‑agnostic, perceptual‑quality ISP built from five lightweight conventional functions, a compact encoder, and a controller that dynamically sets ISP parameters. By combining global and local control mechanisms, it achieves per‑image and per‑region parameterization, enabling a universal ISP that works across unknown sensors. The training strategy includes automatic tuning of parameter search spaces and a denoiser‑as‑denoiser approach with a Local L1 loss to stabilize color fidelity, yielding robust performance. Empirical results on FiveK (universal ISP and enhancement), HDR+ (tone mapping), and MAI21 (normal ISP) demonstrate state‑of‑the‑art perceptual quality with substantially lower computational cost than large DNN ISPs, highlighting practical impact for edge devices and cloud pipelines and suggesting new directions for color‑mapping in DNNs.

Abstract

Full DNN-based image signal processors (ISPs) have been actively studied and have achieved superior image quality compared to conventional ISPs. In contrast to this trend, we propose a lightweight ISP that consists of simple conventional ISP functions but achieves high image quality by increasing expressiveness. Specifically, instead of tuning the parameters of the ISP, we propose to control them dynamically for each environment and even locally. As a result, state-of-the-art accuracy is achieved on various datasets, including other tasks like tone mapping and image enhancement, even though ours is lighter than DNN-based ISPs. Additionally, our method can process different image sensors with a single ISP through dynamic control, whereas conventional methods require training for each sensor.
Paper Structure (21 sections, 6 equations, 9 figures, 5 tables, 2 algorithms)

This paper contains 21 sections, 6 equations, 9 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. ISP
  4. Dynamic Control and Local Parameter Tuning of ISP
  5. Methodology
  6. ISP
  7. Encoder
  8. Controller
  9. Training Methods
  10. Experiments
  11. Datasets
  12. Implementation Details
  13. Evaluation on FiveK Dataset
  14. Evaluation of the Tone Mapping Task Including Low-light Enhancement on HDR+ Dataset
  15. Evaluation of the Normal ISP Task on MAI21 Dataset
  16. ...and 6 more sections

Figures (9)

  • Figure 1: (a) The conventional ISPs tune the parameters of classical ISP functions for each image sensor, while (b) the fully DNN-based ISPs train DNN for each image sensor. (c) Our method handles any sensors by controlling the parameters of classical ISP functions for each environment and each sensor, and even locally.
  • Figure 2: The proposed light weight denoiser.
  • Figure 3: The proposed (a) light weight encoder consists of three (b) blocks modified from the SYENet block gou2023syenet.
  • Figure 4: The proposed local controller. In practical implementation, we don't iterate per region but compute at once using point-wise convolutions within $f_{dec,l}$ and $f_{up,l}$.
  • Figure 5: Visualization of the univeral ISP task on FiveK bychkovsky2011learning. Our method estimates the true color well without artefacts. (Please zoom in.)
  • ...and 4 more figures