Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Quanta Color Imaging

Vishal Purohit, Junjie Luo, Yiheng Chi, Qi Guo, Stanley H. Chan, Qiang Qiu

TL;DR

The core innovation of this paper is an exposure synthesis model framed under a neural ordinary differential equation (Neural ODE) that allows us to generate a contin-uum of exposures from a single observation, resulting in notably enhanced colorization.

Abstract

The astonishing development of single-photon cameras has created an unprecedented opportunity for scientific and industrial imaging. However, the high data throughput generated by these 1-bit sensors creates a significant bottleneck for low-power applications. In this paper, we explore the possibility of generating a color image from a single binary frame of a single-photon camera. We evidently find this problem being particularly difficult to standard colorization approaches due to the substantial degree of exposure variation. The core innovation of our paper is an exposure synthesis model framed under a neural ordinary differential equation (Neural ODE) that allows us to generate a continuum of exposures from a single observation. This innovation ensures consistent exposure in binary images that colorizers take on, resulting in notably enhanced colorization. We demonstrate applications of the method in single-image and burst colorization and show superior generative performance over baselines. Project website can be found at https://vishal-s-p.github.io/projects/2023/generative_quanta_color.html.

Generative Quanta Color Imaging

TL;DR

The core innovation of this paper is an exposure synthesis model framed under a neural ordinary differential equation (Neural ODE) that allows us to generate a contin-uum of exposures from a single observation, resulting in notably enhanced colorization.

Abstract

The astonishing development of single-photon cameras has created an unprecedented opportunity for scientific and industrial imaging. However, the high data throughput generated by these 1-bit sensors creates a significant bottleneck for low-power applications. In this paper, we explore the possibility of generating a color image from a single binary frame of a single-photon camera. We evidently find this problem being particularly difficult to standard colorization approaches due to the substantial degree of exposure variation. The core innovation of our paper is an exposure synthesis model framed under a neural ordinary differential equation (Neural ODE) that allows us to generate a continuum of exposures from a single observation. This innovation ensures consistent exposure in binary images that colorizers take on, resulting in notably enhanced colorization. We demonstrate applications of the method in single-image and burst colorization and show superior generative performance over baselines. Project website can be found at https://vishal-s-p.github.io/projects/2023/generative_quanta_color.html.
Paper Structure (14 sections, 1 theorem, 5 equations, 9 figures, 2 tables)

This paper contains 14 sections, 1 theorem, 5 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Approach
  4. One-Bit Image Formation Model
  5. Why Exposure Synthesis is Required for Colorization?
  6. Continuous Exposure Synthesis using Filter Decomposed Neural ODEs
  7. Exposure Adaptive Colorization
  8. Experiments
  9. Experimental Setup
  10. Exposure Burst Recovery Results
  11. Single and Burst Image Colorization Results
  12. Real Data Experiments
  13. Benefits for Practical Applications
  14. Conclusion

Key Result

Theorem 3.1

If $\mathbf{\Lambda}_1$ and $\mathbf{\Lambda}_2$ are filter atoms generated with neural ODE using integration intervals ($\widetilde{\theta}_{\text{0}}$, $\widetilde{\theta}_{\text{1}}$) and ($\widetilde{\theta}_0,\widetilde{\theta}_2$), respectively such that $\|\mathbf{\Lambda}_1 - \mathbf{\Lambda

Figures (9)

  • Figure 1: We introduce generative quanta color imaging. Given a binary frame captured by a single-photon camera (quanta image sensor (QIS) in this example), the proposed method generates a continuum of exposures using a neural ordinary differential equation framework. Color are then generated based on these exposures. Left: A qualitative comparison between our approach and existing methods is shown. Right: Comparison between colorization results and image captured using a RGB CMOS camera under similar conditions.
  • Figure 2: Illustration of exposure correction and colorization of binary images using neural networks. (a) depicts a range of images from overexposed to underexposed, illustrating the degradation of image details due to exposure variation. (b) contrasts the standard colorization workflow and our proposed approach. (i) In standard colorization approaches, a neural network learns to map a binary image $\mathbf{Y}$ to the corresponding color image $\mathbf{X}_{c}$ via a neural network, $\mathcal{F}^{\text{aug}}$, where superscript 'aug' indicates the colorizer is trained using dataset with augmented exposure images. (ii) In contrast, our approach does not require training colorizer with augmented exposure images. (c) compares the colorization results: the first row is the output of a colorizer trained without augmentation, the second row is the output of colorizer trained with augmented data, the third row corresponds to the results of our method and the last provides the ground truth images for reference.
  • Figure 3: An illustration of our proposed method for exposure adaptive colorization: A binary image, $\mathbf{Y}$, which can be overexposed or underexposed, is input into the proposed exposure synthesis module. The colorization of binary image can be achieved using Single Image Colorization (SIC) or Burst Image Colorization (BIC). (i) SIC: Based on the input and target exposure levels, $\widetilde{\theta}_\text{input}$ and $\widetilde{\theta}_\text{target}$, this module adjusts the weights of a exposure synthesis network $\mathcal{G}$, which then generates an exposure-corrected image. Note that corrected image is not necessarily a binary image. Since the colorization module $\mathcal{F}$ is trained to colorize only image of specific exposure, the exposure synthesis module ensures corrected binary image has similar exposure to the one on which $\mathcal{F}$ is trained. (ii) BIC: For BIC we generate images with varying exposures as input to the burst image colorization network, $\mathcal{F}_{\text{burst}}$. The trained network is able to exploit the complementary information across multiple exposures with the help of Cross Non-Local Fusion blocks Luo_2021_CVPR to synthesize colors in regions of the image that is otherwise not possible by SIC approach.
  • Figure 4: Qualitative results for exposure burst recovery experiments for AFHQ dataset. We compare our methods, AtomODE-Pix2Pix and AtomODE-CycleGAN, with DLOW, SAVI2I and DNI.
  • Figure 5: Colorized results for AFHQ dataset using various combinations of exposure correction methods and colorizer. The first column shows the overexposed binary input image for exposure correction. The results of our methods AtomODE Pix2Pix and AtomODE CycleGAN are shown in columns six and seven.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 3.1