Zero-Shot Image Denoising for High-Resolution Electron Microscopy

TL;DR

Noise2SR, a zero-shot self-supervised learning (ZS-SSL) denoising framework for HREM, is proposed, which outperforms state-of-the-art ZS-SSL methods and achieves comparable denoising performance with supervised methods and suggests its potential for improving the SNR of images in material imaging domains.

Abstract

High-resolution electron microscopy (HREM) imaging technique is a powerful tool for directly visualizing a broad range of materials in real-space. However, it faces challenges in denoising due to ultra-low signal-to-noise ratio (SNR) and scarce data availability. In this work, we propose Noise2SR, a zero-shot self-supervised learning (ZS-SSL) denoising framework for HREM. Within our framework, we propose a super-resolution (SR) based self-supervised training strategy, incorporating the Random Sub-sampler module. The Random Sub-sampler is designed to generate approximate infinite noisy pairs from a single noisy image, serving as an effective data augmentation in zero-shot denoising. Noise2SR trains the network with paired noisy images of different resolutions, which is conducted via SR strategy. The SR-based training facilitates the network adopting more pixels for supervision, and the random sub-sampling helps compel the network to learn continuous signals enhancing the robustness. Meanwhile, we mitigate the uncertainty caused by random-sampling by adopting minimum mean squared error (MMSE) estimation for the denoised results. With the distinctive integration of training strategy and proposed designs, Noise2SR can achieve superior denoising performance using a single noisy HREM image. We evaluate the performance of Noise2SR in both simulated and real HREM denoising tasks. It outperforms state-of-the-art ZS-SSL methods and achieves comparable denoising performance with supervised methods. The success of Noise2SR suggests its potential for improving the SNR of images in material imaging domains.

Figures (15)

• Figure 1: Pipeline of proposed Noise2SR framework. A. Training Phase: First, Random Sub-sampler takes a noisy image $\mathbf{y}$ as input and generates a sub-sampled noisy image $\mathbf{y}_{J^\complement}$ along with corresponding unsampled mask $\mathbf{m}_J$. Then, the network $f_\theta$ takes the sub-sampled image $\mathbf{y}_{J^\complement}$ as input and generates a denoised image of full resolution $f_\theta(\mathbf{y}_{J^\complement})$. The network is optimized by computing the loss on the difference between unsampled noisy pixels $\mathbf{y}_J$ and the output of the network. B. Inference Phase: A sub-sampled noisy set $\mathcal{Y}$ can be obtained by repeatedly sub-sampling a noisy image $\mathbf{y}$$M$ times using the Random Sub-sampler. Given a sub-sampled noisy set $\small{\mathcal{Y}}$, well-trained network $f_{\hat{\theta}}$ can generated a plausible denoised image set $\small{\hat{\mathcal{X}}}$. Finally, the clean image can be estimated by averaging the images in $\small{\hat{\mathcal{X}}}$ using the MMSE estimation.
• Figure 2: Illustration of the operations in the Random Sub-sampler with sampling stride $s$ to generate sub-sampled image $\mathbf{y}_{J^\complement}$ and unsampled mask $\mathbf{m}_J$. First, the input image is applied pixel unshuffling with a stride of $s$. At each location in the pixel unshuffled image, the Random Sub-sampler randomly selects $1$ element along the channel dimension to compose the sub-sampled image $\mathbf{y}_{J^\complement}$. Meanwhile, the sampler sets the unsampled pixel mask $\mathbf{m}_J$ to 0 at the corresponding location if the element has been selected. Otherwise, it assigns a value of 1 (where black represents 0, and white represents 1). The specific example in the figure demonstrates the sub-sampling process of the Random Sub-sampler with sampling stride $(s = 2)$ applied to an input image of size $4 \times 4$.
• Figure 3: The architecture of Noise2SR used for parameterizing the super-resolved denoising function $f_\theta$, which consists of the U-Net based encoder and thesuper-resolved decoder.
• Figure 4: Comparison of Noise2SR and other image denoising methods on simulated Pe/CeO2 catalyst corrupted with Poisson-Gaussian noise $(a = 0.05, b = 0.02)$. The second and fourth rows display the corresponding error maps of the denoised results. * indicates that the dataset-based self-supervised denoising method was performed in a zero-shot learning manner.
• Figure 5: Comparison of intensity profiles on the surface atomic columns is conducted for the denoised results obtained using FBI-Denoiser (FBI-D) and Noise2SR (N2SR) on simulated Pe/CeO2 catalyst corrupted with Poisson-Gaussian noise $(a = 0.05, b = 0.02)$, alongside the corresponding ground truth data.