Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revealing Microscopic Objects in Fluorescence Live Imaging by Video-to-video Translation Based on A Spatial-temporal Generative Adversarial Network

Yang Jiao, Mei Yang, Mo Weng

TL;DR

This work tackles the limitation of fluorescence microscopy where spectral cross-talk restricts the number of observable fluorophores. It introduces STGAN, a spatial-temporal GAN-based framework for cross-domain video-to-video translation in microscopy, leveraging both spatial and temporal cues to translate between object domains without additional labeling. The model combines adversarial, spatial reconstruction, and temporal reconstruction losses to learn robust mappings and produces six output variants to enable flexible visualization. Experimental results on live-cell data show STGAN outperforms baselines and can simulate the spatial distribution of one protein from another, offering a practical approach to visualize multiple components in fluorescence imaging.

Abstract

In spite of being a valuable tool to simultaneously visualize multiple types of subcellular structures using spectrally distinct fluorescent labels, a standard fluoresce microscope is only able to identify a few microscopic objects; such a limit is largely imposed by the number of fluorescent labels available to the sample. In order to simultaneously visualize more objects, in this paper, we propose to use video-to-video translation that mimics the development process of microscopic objects. In essence, we use a microscopy video-to-video translation framework namely Spatial-temporal Generative Adversarial Network (STGAN) to reveal the spatial and temporal relationships between the microscopic objects, after which a microscopy video of one object can be translated to another object in a different domain. The experimental results confirm that the proposed STGAN is effective in microscopy video-to-video translation that mitigates the spectral conflicts caused by the limited fluorescent labels, allowing multiple microscopic objects be simultaneously visualized.

Revealing Microscopic Objects in Fluorescence Live Imaging by Video-to-video Translation Based on A Spatial-temporal Generative Adversarial Network

TL;DR

This work tackles the limitation of fluorescence microscopy where spectral cross-talk restricts the number of observable fluorophores. It introduces STGAN, a spatial-temporal GAN-based framework for cross-domain video-to-video translation in microscopy, leveraging both spatial and temporal cues to translate between object domains without additional labeling. The model combines adversarial, spatial reconstruction, and temporal reconstruction losses to learn robust mappings and produces six output variants to enable flexible visualization. Experimental results on live-cell data show STGAN outperforms baselines and can simulate the spatial distribution of one protein from another, offering a practical approach to visualize multiple components in fluorescence imaging.

Abstract

In spite of being a valuable tool to simultaneously visualize multiple types of subcellular structures using spectrally distinct fluorescent labels, a standard fluoresce microscope is only able to identify a few microscopic objects; such a limit is largely imposed by the number of fluorescent labels available to the sample. In order to simultaneously visualize more objects, in this paper, we propose to use video-to-video translation that mimics the development process of microscopic objects. In essence, we use a microscopy video-to-video translation framework namely Spatial-temporal Generative Adversarial Network (STGAN) to reveal the spatial and temporal relationships between the microscopic objects, after which a microscopy video of one object can be translated to another object in a different domain. The experimental results confirm that the proposed STGAN is effective in microscopy video-to-video translation that mitigates the spectral conflicts caused by the limited fluorescent labels, allowing multiple microscopic objects be simultaneously visualized.

Paper Structure

This paper contains 13 sections, 15 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Problem definition of cross-domain translation
  4. Framework
  5. Objectives
  6. Adversarial losses
  7. Spatial reconstruction losses
  8. Temporal reconstruction losses
  9. Full objectives
  10. STGAN outputs
  11. Experiment
  12. Results
  13. Conclusion

Figures (3)

  • Figure 1: STGAN framework. $\tilde{u_t}$ and $\tilde{v_t}$ are outputs of spatial generators. $\tilde{u_t}=F_s(v_t)$ and $\tilde{v_t}=G_s(u_t)$. $\dot{u_t}$ and $\dot{v_t}$ are outputs of temporal generators based on input frames. $\dot{u_t}=F_t(u_{1:\tau-1})$ and $\dot{v_t}=G_t(v_{1:\tau-1})$. $\ddot{u_\tau}$ and $\ddot{v_\tau}$ are outputs of temporal generators based on fake frames. $\ddot{u_\tau} = F_t(\hat{u_{1:\tau-1}})$ and $\ddot{v_\tau} = G_t(\hat{v_{1:\tau-1}})$
  • Figure 2: Cross-domain translation results of STGAN. $u(t)$ is a frame of Myosin live image. $v(t)$ is a frame of Ajuba live image. $t$ is the time stamp of the live image.
  • Figure 3: Cross-domain translation evaluation.