No-Clean-Reference Image Super-Resolution: Application to Electron Microscopy

TL;DR

This work tackles the challenge of obtaining clean, high-resolution 3D electron microscopy images over large brain volumes by proposing EMSR, a deep-learning SR framework that can be trained with no-clean references. EMSR integrates edge-attention and Vision Transformer blocks with a noise-robust, self-supervised design and uses a multi-term loss to learn from corrupted LR/HR pairs as well as denoised references. Through experiments on nine brain datasets, EMSR demonstrates competitive or superior performance to several SR baselines, with the ability to train effectively using real, denoised, or synthetically degraded pairs. The approach enables faster, less costly EM imaging while providing high-quality reconstructions suitable for downstream neuroanatomical analysis, and highlights the potential of synthetic degradations to augment EM SR under matched conditions.

Abstract

The inability to acquire clean high-resolution (HR) electron microscopy (EM) images over a large brain tissue volume hampers many neuroscience studies. To address this challenge, we propose a deep-learning-based image super-resolution (SR) approach to computationally reconstruct clean HR 3D-EM with a large field of view (FoV) from noisy low-resolution (LR) acquisition. Our contributions are I) Investigating training with no-clean references for $\ell_2$ and $\ell_1$ loss functions; II) Introducing a novel network architecture, named EMSR, for enhancing the resolution of LR EM images while reducing inherent noise; and, III) Comparing different training strategies including using acquired LR and HR image pairs, i.e., real pairs with no-clean references contaminated with real corruptions, the pairs of synthetic LR and acquired HR, as well as acquired LR and denoised HR pairs. Experiments with nine brain datasets showed that training with real pairs can produce high-quality super-resolved results, demonstrating the feasibility of training with non-clean references for both loss functions. Additionally, comparable results were observed, both visually and numerically, when employing denoised and noisy references for training. Moreover, utilizing the network trained with synthetically generated LR images from HR counterparts proved effective in yielding satisfactory SR results, even in certain cases, outperforming training with real pairs. The proposed SR network was compared quantitatively and qualitatively with several established SR techniques, showcasing either the superiority or competitiveness of the proposed method in mitigating noise while recovering fine details.

Figures (12)

• Figure 1: Schematic diagram of serial block-face scanning electron microscopy and imaging. a) The electron gun generates streams of electrons that are focused and raster scanned across the sample's surface (solid yellow lines). The interaction of these focused electrons with the sample results in the ejection of electron streams (dashed yellow lines), which are collected by detectors to form a 2D image of $k$-th slice (labeled as $\small{\#}k$). Note that the region of interest from the sample is imaged at LR with a large FoV (marked in orange), while at HR, the FoV is smaller (marked in red). After imaging a slice, the diamond knife is used to cut the sample to a specific thickness, determining the resolution in the $z$ direction and exposing the subsequent block-face for imaging. Imperfections in the imaging device components can introduce blurring and noise in the resultant images (solid green arrows). b) Stack of 2D imaged slices constitutes the 3D-EM dataset. c) LR 3D-EM corresponding to d) HR 3D-EM from small FoV. The zoomed-in area from (c) and (d) demonstrates the superior quality of the HR image in terms of contrast and resolution, see asterisks.
• Figure 2: Overview of the proposed image super-resolution network for training with pairs of corrupted images. The network includes the feature extractor, edge attention, and reconstruction modules, which are shared between the original noisy LR EM image $y$ and its noisier version $y^{\prime}$. The network is encouraged to generate two outputs, $f_{\theta}(y)$ and $f_{\theta}(y^{\prime})$, that are consistent with the noisy reference image $x$. The output from the original image $f_{\theta}(y)$ serves as the reference for the noisier-noisy input, establishing a noise-robust framework in a self-supervised approach.
• Figure 3: Modules embedded in the proposed network: Basic Block, Residual Block, À-Trous Wavelet, Attention Block, and Vision Transformer Block.
• Figure 4: Multi-scale edges extracted from the EM dataset using ATW, where $h$ represents the filter's kernel. The figure illustrates three edge components obtained at different scales, demonstrating the sparsity of edges and underscoring the importance of paying attention to edge details.
• Figure 5: Visual comparisons of super-resolution methods for BRAIN5[IPSI] are presented, and magnified regions are presented to facilitate comparison.