Predicting fluorescent labels in label-free microscopy images with pix2pix and adaptive loss in Light My Cells challenge

TL;DR

This paper proposes a deep learning-based in silico labeling method for the Light My Cells challenge, built upon pix2pix, and explores the effectiveness of several training strategies to handle different input modalities.

Abstract

Fluorescence labeling is the standard approach to reveal cellular structures and other subcellular constituents for microscopy images. However, this invasive procedure may perturb or even kill the cells and the procedure itself is highly time-consuming and complex. Recently, in silico labeling has emerged as a promising alternative, aiming to use machine learning models to directly predict the fluorescently labeled images from label-free microscopy. In this paper, we propose a deep learning-based in silico labeling method for the Light My Cells challenge. Built upon pix2pix, our proposed method can be trained using the partially labeled datasets with an adaptive loss. Moreover, we explore the effectiveness of several training strategies to handle different input modalities, such as training them together or separately. The results show that our method achieves promising performance for in silico labeling. Our code is available at https://github.com/MedICL-VU/LightMyCells.

Figures (6)

• Figure 1: Problem formulation of the challenge. The goal of the challenge is to predict fluorescently labeled images for four organelles (output) from label-free transmitted light microscopy images (input). The input images may have different modalities, (i.e., BF, PC or DIC) and the labels for certain organelles may not be available.
• Figure 2: Dataset overview. The challenge dataset consists of 30 sub-datasets collected from different studies. (a): The available input modalities and labeled organelles for all 30 studies. (b): The labeled organelles for each input modality (from top to bottom: DIC, BF and PC). M: mitochondria, N: nucleus, T: tubulin, A: actin.
• Figure 3: Network architecture of our modified pix2pix model.
• Figure 4: The illustration of the adaptive loss for partial label training. The network prediction $\hat{y}$ and the ground truth $y$ are transformed by removing the unlabeled organelles such that their predictions are excluded from loss computation.
• Figure 5: Three training strategies to handle different input modalities.