Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAN with Skip Patch Discriminator for Biological Electron Microscopy Image Generation

Nishith Ranjon Roy, Nailah Rawnaq, Tulin Kaman

TL;DR

The paper addresses the challenge of generating realistic electron microscopy images under limited data by introducing a skip patch discriminator that provides multi-scale patch access, enabling simultaneous modeling of global and local structures. It employs a two-stage pipeline where masks are first generated from noise and then used to synthesize EM images, comparing a patch-based discriminator, a 16×16/70×70 patch mix, and the proposed skip patch approach within a cGAN framework. Key contributions include improved realism and faster convergence (about half the training epochs) with the skip patch discriminator, and demonstrated effectiveness on a Drosophila EM dataset focusing on mitochondria and cell membranes. The findings support using skip patch discrimination for EM and other high-resolution, structurally complex biological images, particularly when data are scarce, to reduce artifacts and improve training efficiency.

Abstract

Generating realistic electron microscopy (EM) images has been a challenging problem due to their complex global and local structures. Isola et al. proposed pix2pix, a conditional Generative Adversarial Network (GAN), for the general purpose of image-to-image translation; which fails to generate realistic EM images. We propose a new architecture for the discriminator in the GAN providing access to multiple patch sizes using skip patches and generating realistic EM images.

GAN with Skip Patch Discriminator for Biological Electron Microscopy Image Generation

TL;DR

The paper addresses the challenge of generating realistic electron microscopy images under limited data by introducing a skip patch discriminator that provides multi-scale patch access, enabling simultaneous modeling of global and local structures. It employs a two-stage pipeline where masks are first generated from noise and then used to synthesize EM images, comparing a patch-based discriminator, a 16×16/70×70 patch mix, and the proposed skip patch approach within a cGAN framework. Key contributions include improved realism and faster convergence (about half the training epochs) with the skip patch discriminator, and demonstrated effectiveness on a Drosophila EM dataset focusing on mitochondria and cell membranes. The findings support using skip patch discrimination for EM and other high-resolution, structurally complex biological images, particularly when data are scarce, to reduce artifacts and improve training efficiency.

Abstract

Generating realistic electron microscopy (EM) images has been a challenging problem due to their complex global and local structures. Isola et al. proposed pix2pix, a conditional Generative Adversarial Network (GAN), for the general purpose of image-to-image translation; which fails to generate realistic EM images. We propose a new architecture for the discriminator in the GAN providing access to multiple patch sizes using skip patches and generating realistic EM images.
Paper Structure (7 sections, 2 equations, 3 figures)

This paper contains 7 sections, 2 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Dataset (Drosophila)
  3. Methodology
  4. Skip patch discriminator
  5. Mitochondria and cell membrane mask (Fig. \ref{['fig:figure1_comb']}-A.i)
  6. EM Image of cellular structure (Fig. \ref{['fig:figure1_comb']}-A.ii)
  7. Results and conclusions

Figures (3)

  • Figure 1: (A) The process of generating masks ($X$) and EM images ($Y$). The generator and discriminator are denoted by $G$ and $D$ respectively. The first stage of this process involves generating masks (iii) from random noise ($z$), while the second stage involves using these generated masks to produce EM images (iv). (B) The principal component analysis (PCA) plot shows the similarity among the distribution of the generated mask with the real mask. The red points comparatively far on the right side indicate that the generated mask with $16\times 16$ patch does not show any alignment with the true mask. On the contrary, the mixture in the center of three different points suggests that the generated mask with $70\times70$ patch, skip patch (ours), and real mask have approximately the same type of distribution. (C) The four images show the real and generated masks via different methods. A fixed 6000 training iterations (or epochs) has been performed for all three methods.
  • Figure 2: (A) The discriminator architecture proposed by goodfellow2014generative. (B) The new discriminator architecture proposed in this paper. (C) U-Net architecture for the generator in the conditional GAN (cGAN) EM images generation part.
  • Figure 3: (A) Outcome of the conditional GAN (cGAN) models with the training data where S.Patch, P(16x16) and P(70x70) correspond to the skip-patch, $16\times 16$ patch and $70\times 70$ patch discriminators respectively. (B) The outcome of the cGAN models with the newly generated mask (zoomed). All of the points mix evenly in the PCA plot of the generated images vs the real image. Hence according to PCA, the distribution of the generated images with all of the three methods has a distribution close to the real EM images.