Multi-scale Conditional Generative Modeling for Microscopic Image Restoration

TL;DR

A multi-scale generative model that enhances conditional image restoration through a novel exploitation of the Brownian Bridge process within wavelet domain that provides significant acceleration during training and sampling while sustaining a high image generation quality and diversity on par with SOTA diffusion models.

Abstract

The advance of diffusion-based generative models in recent years has revolutionized state-of-the-art (SOTA) techniques in a wide variety of image analysis and synthesis tasks, whereas their adaptation on image restoration, particularly within computational microscopy remains theoretically and empirically underexplored. In this research, we introduce a multi-scale generative model that enhances conditional image restoration through a novel exploitation of the Brownian Bridge process within wavelet domain. By initiating the Brownian Bridge diffusion process specifically at the lowest-frequency subband and applying generative adversarial networks at subsequent multi-scale high-frequency subbands in the wavelet domain, our method provides significant acceleration during training and sampling while sustaining a high image generation quality and diversity on par with SOTA diffusion models. Experimental results on various computational microscopy and imaging tasks confirm our method's robust performance and its considerable reduction in its sampling steps and time. This pioneering technique offers an efficient image restoration framework that harmonizes efficiency with quality, signifying a major stride in incorporating cutting-edge generative models into computational microscopy workflows.

Figures (14)

• Figure 1: Inherent information loss in autoencoder backbones. The same high-quality image passes the three generative models without going through the diffusion process. Public pre-trained checkpoints are used for LDM ldm and Refusion luo2023refusion. LDM and Refusion suffer from reconstruction loss due to lossy autoencoder backbones while the wavelet and inverse wavelet transform pairs are lossless.
• Figure 2: Schematic diagram of MSCGM. The conditional image $y_L^0$ is first decomposed by multi-scale wavelet transform (WT). In the coarsest wavelet layer, a BBDP transforms the low-frequency subband of conditional image to the low-frequency subband of the target image. A multi-scale GAN transforms subsequent high-frequency subbands of conditional image to the high-frequency subbands of the target image and recovers the full-resolution image using inverse wavelet transform (IWT). $y_L^i$ and $y_H^i$ represent the low- and high-frequency wavelet coefficients of the conditional image at the $i^{th}$ level of wavelet transform, respectively. Similarly, $x_L^i$ and $x_H^i$ denote the low-and high-frequency wavelet coefficients of the target image at the $i^{th}$ level of wavelet transform.
• Figure 3: Comparison of CMSR wang2019deep and our method on microscopy image super-resolution of nanobeads. (a) Input images captured by a confocal microscope, (b, c) SR outputs of CMSR and our method, and (d) ground truths captured by an STED microscope of the same FOV. (e-l) Zoom-in regions marked by the corresponding white boxes in (a-c). Cross-section intensity values along the dashed line are plotted.
• Figure 4: Microscopy image super-resolution of our method on HeLa cells. (a) LR confocal image, (b) SR output image and (c) HR STED image of the same FOV.
• Figure 5: KL divergence between the standard normal distribution and normalized sample distribution with respect to the wavelet scale. Images were sampled from DIV2K dataset by $32\times 32$ and $64\times 64$ patches.

Theorems & Definitions (10)

• Theorem 1
• Definition 2: Forward Brownian bridge process
• Theorem 3
• Definition 4: Multi-scale wavelet decomposition of conditional image generation
• Theorem 5
• Proposition 1
• Theorem 6
• Theorem 7
• Proposition 2
• proof