Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modality Microscopy Image Style Transfer for Nuclei Segmentation

Ye Liu, Sophia J. Wagner, Tingying Peng

TL;DR

With the authors' style augmentation, the segmentation accuracy of the two top-ranked Mask R-CNN-based nuclei segmentation algorithms in the competition increases significantly and helps counteract class imbalance without resampling of minority classes.

Abstract

Annotating microscopy images for nuclei segmentation is laborious and time-consuming. To leverage the few existing annotations, also across multiple modalities, we propose a novel microscopy-style augmentation technique based on a generative adversarial network (GAN). Unlike other style transfer methods, it can not only deal with different cell assay types and lighting conditions, but also with different imaging modalities, such as bright-field and fluorescence microscopy. Using disentangled representations for content and style, we can preserve the structure of the original image while altering its style during augmentation. We evaluate our data augmentation on the 2018 Data Science Bowl dataset consisting of various cell assays, lighting conditions, and imaging modalities. With our style augmentation, the segmentation accuracy of the two top-ranked Mask R-CNN-based nuclei segmentation algorithms in the competition increases significantly. Thus, our augmentation technique renders the downstream task more robust to the test data heterogeneity and helps counteract class imbalance without resampling of minority classes.

Multi-Modality Microscopy Image Style Transfer for Nuclei Segmentation

TL;DR

With the authors' style augmentation, the segmentation accuracy of the two top-ranked Mask R-CNN-based nuclei segmentation algorithms in the competition increases significantly and helps counteract class imbalance without resampling of minority classes.

Abstract

Annotating microscopy images for nuclei segmentation is laborious and time-consuming. To leverage the few existing annotations, also across multiple modalities, we propose a novel microscopy-style augmentation technique based on a generative adversarial network (GAN). Unlike other style transfer methods, it can not only deal with different cell assay types and lighting conditions, but also with different imaging modalities, such as bright-field and fluorescence microscopy. Using disentangled representations for content and style, we can preserve the structure of the original image while altering its style during augmentation. We evaluate our data augmentation on the 2018 Data Science Bowl dataset consisting of various cell assays, lighting conditions, and imaging modalities. With our style augmentation, the segmentation accuracy of the two top-ranked Mask R-CNN-based nuclei segmentation algorithms in the competition increases significantly. Thus, our augmentation technique renders the downstream task more robust to the test data heterogeneity and helps counteract class imbalance without resampling of minority classes.
Paper Structure (12 sections, 1 equation, 5 figures, 1 table)

This paper contains 12 sections, 1 equation, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Dataset description
  4. Data augmentation step I: clustering training images into different modalities
  5. Data augmentation step II: multi-modality style transfer
  6. Mask R-CNN for instance segmentation
  7. Test time augmentation
  8. Results and Discussion
  9. Clustering into modalities
  10. Improved segmentation performance by style transfer augmentation
  11. Conclusion
  12. Compliance with Ethical Standards

Figures (5)

  • Figure 1: Overview of the nuclei segmentation pipeline with multi-modality style transfer for data augmentation.
  • Figure 2: (a) PCA decomposition of hue saturation values of all training images and their class distribution. Each point represents one image, colored according to the corresponding cluster. (b) Exemplary images for each cluster.
  • Figure 3: Examples of the outcome of multi-modality style transfer. First column: original images. Remaining columns: result of each domain's multi-modality style transfer with self-reconstruction on the diagonal, highlighted in blue boxes.
  • Figure 4: Segmentation results without (first row) and with (second row) our augmentation. Predicted mask contours in common are shown in blue. Yellow contours in the second row show the improvements compared to the first row.
  • Figure 5: The style transfer GAN fails when images at two modalities contain different content.