Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder

Dihan Zheng, Yihang Zou, Xiaowen Zhang, Chenglong Bao

TL;DR

This study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages both paired and un-paired datasets to generate realistic degraded data and demonstrates remarkable performance in downstream image restoration tasks, even with limited paired data.

Abstract

The data bottleneck has emerged as a fundamental challenge in learning based image restoration methods. Researchers have attempted to generate synthesized training data using paired or unpaired samples to address this challenge. This study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages both paired and unpaired datasets to generate realistic degraded data. Our approach is based on modeling the conditional distribution of degraded and clean images with a specially designed graphical model. Under the variational inference framework, we develop an objective function for handling both paired and unpaired data. We employ our method to generate paired training samples for real-world image denoising and super-resolution tasks. Our approach excels in the quality of synthetic degraded images compared to other unpaired and paired noise modeling methods. Furthermore, our approach demonstrates remarkable performance in downstream image restoration tasks, even with limited paired data. With more paired data, our method achieves the best performance on the SIDD dataset.

SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder

TL;DR

This study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages both paired and un-paired datasets to generate realistic degraded data and demonstrates remarkable performance in downstream image restoration tasks, even with limited paired data.

Abstract

The data bottleneck has emerged as a fundamental challenge in learning based image restoration methods. Researchers have attempted to generate synthesized training data using paired or unpaired samples to address this challenge. This study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages both paired and unpaired datasets to generate realistic degraded data. Our approach is based on modeling the conditional distribution of degraded and clean images with a specially designed graphical model. Under the variational inference framework, we develop an objective function for handling both paired and unpaired data. We employ our method to generate paired training samples for real-world image denoising and super-resolution tasks. Our approach excels in the quality of synthetic degraded images compared to other unpaired and paired noise modeling methods. Furthermore, our approach demonstrates remarkable performance in downstream image restoration tasks, even with limited paired data. With more paired data, our method achieves the best performance on the SIDD dataset.
Paper Structure (12 sections, 1 theorem, 22 equations, 7 figures, 7 tables)

This paper contains 12 sections, 1 theorem, 22 equations, 7 figures, 7 tables.

Table of Contents

  1. Our methodology
  2. Proposed model
  3. Model settings
  4. Experiment and results
  5. Datasets
  6. Implementation details
  7. Noise synthesis
  8. Downstream denoising
  9. Finetune denoising network
  10. Downstream SR
  11. Ablation study and discussions
  12. Conclusion

Key Result

Proposition 1

Let $q({\mathbf{z}}|{\mathbf{x}},{\mathbf{y}})$ be a mixture model of $q({\mathbf{z}}|{\mathbf{x}})$ and $q({\mathbf{z}}|{\mathbf{y}})$, as described in mix_model, then: Moreover, suppose that $q({\mathbf{z}}|{\mathbf{x}})=p({\mathbf{z}}|{\mathbf{x}})$ by sharing the same neural network. Then:

Figures (7)

  • Figure 1: a: generation process for $({\mathbf{x}}, {\mathbf{y}})$ and the corresponding inference model. b: degradation generation procedure.
  • Figure 2: Data flow of the proposed semi-supervised noise modeling method that models three kinds of data: paired domain (degraded-clean image pairs), source domain (only clean images), and target domain (only degraded images).
  • Figure 3: The hierarchical structure and network architecture.
  • Figure 4: Visual comparison of generated noisy images, "PD" denotes paired data.
  • Figure 5: Visual comparison of denoising performance.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Proposition 1