Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches

Shawn Mathew, Saad Nadeem, Alvin C. Goh, Arie Kaufman

TL;DR

The paper tackles the lack of temporally coherent data in frame-based colonoscopy domain translation by introducing RT-GAN, a lightweight approach that adds temporal consistency without full video-model training. It uses a single generator G that takes $(x_t, x_{t-1}, y_{t-1}')$ to produce $y_t'$, guided by a temporal discriminator operating on 3-frame sequences and a spatial discriminator, with a combined objective $ \mathcal{L}_{obj} = \lambda \mathcal{L}_{t} + \mathcal{L}_{f} + \mathcal{L}_{s}$ to balance temporal fidelity and frame-based fidelity. The method reduces training resources by around 5x relative to full video-domain methods and demonstrates improved segmentation and realistic video generation on colonoscopy tasks, including haustral fold segmentation and synthetic-to-real video synthesis; a temporal colonoscopy dataset and code are released for reproducibility. This work offers a practical path to temporally coherent frame-based translation in medical endoscopy, enabling more reliable AI-assisted analysis and documentation without prohibitive computational cost.

Abstract

Fourteen million colonoscopies are performed annually just in the U.S. However, the videos from these colonoscopies are not saved due to storage constraints (each video from a high-definition colonoscope camera can be in tens of gigabytes). Instead, a few relevant individual frames are saved for documentation/reporting purposes and these are the frames on which most current colonoscopy AI models are trained on. While developing new unsupervised domain translation methods for colonoscopy (e.g. to translate between real optical and virtual/CT colonoscopy), it is thus typical to start with approaches that initially work for individual frames without temporal consistency. Once an individual-frame model has been finalized, additional contiguous frames are added with a modified deep learning architecture to train a new model from scratch for temporal consistency. This transition to temporally-consistent deep learning models, however, requires significantly more computational and memory resources for training. In this paper, we present a lightweight solution with a tunable temporal parameter, RT-GAN (Recurrent Temporal GAN), for adding temporal consistency to individual frame-based approaches that reduces training requirements by a factor of 5. We demonstrate the effectiveness of our approach on two challenging use cases in colonoscopy: haustral fold segmentation (indicative of missed surface) and realistic colonoscopy simulator video generation. We also release a first-of-its kind temporal dataset for colonoscopy for the above use cases. The datasets, accompanying code, and pretrained models will be made available on our Computational Endoscopy Platform GitHub (https://github.com/nadeemlab/CEP). The supplementary video is available at https://youtu.be/UMVP-uIXwWk.

RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches

TL;DR

The paper tackles the lack of temporally coherent data in frame-based colonoscopy domain translation by introducing RT-GAN, a lightweight approach that adds temporal consistency without full video-model training. It uses a single generator G that takes to produce , guided by a temporal discriminator operating on 3-frame sequences and a spatial discriminator, with a combined objective to balance temporal fidelity and frame-based fidelity. The method reduces training resources by around 5x relative to full video-domain methods and demonstrates improved segmentation and realistic video generation on colonoscopy tasks, including haustral fold segmentation and synthetic-to-real video synthesis; a temporal colonoscopy dataset and code are released for reproducibility. This work offers a practical path to temporally coherent frame-based translation in medical endoscopy, enabling more reliable AI-assisted analysis and documentation without prohibitive computational cost.

Abstract

Fourteen million colonoscopies are performed annually just in the U.S. However, the videos from these colonoscopies are not saved due to storage constraints (each video from a high-definition colonoscope camera can be in tens of gigabytes). Instead, a few relevant individual frames are saved for documentation/reporting purposes and these are the frames on which most current colonoscopy AI models are trained on. While developing new unsupervised domain translation methods for colonoscopy (e.g. to translate between real optical and virtual/CT colonoscopy), it is thus typical to start with approaches that initially work for individual frames without temporal consistency. Once an individual-frame model has been finalized, additional contiguous frames are added with a modified deep learning architecture to train a new model from scratch for temporal consistency. This transition to temporally-consistent deep learning models, however, requires significantly more computational and memory resources for training. In this paper, we present a lightweight solution with a tunable temporal parameter, RT-GAN (Recurrent Temporal GAN), for adding temporal consistency to individual frame-based approaches that reduces training requirements by a factor of 5. We demonstrate the effectiveness of our approach on two challenging use cases in colonoscopy: haustral fold segmentation (indicative of missed surface) and realistic colonoscopy simulator video generation. We also release a first-of-its kind temporal dataset for colonoscopy for the above use cases. The datasets, accompanying code, and pretrained models will be made available on our Computational Endoscopy Platform GitHub (https://github.com/nadeemlab/CEP). The supplementary video is available at https://youtu.be/UMVP-uIXwWk.
Paper Structure (5 sections, 5 equations, 4 figures, 2 tables)

This paper contains 5 sections, 5 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. RT-GAN: Recurrent Temporal GAN
  3. Results and Discussion
  4. Acknowledgments.
  5. Disclosure of Interests.

Figures (4)

  • Figure 1: Depicting how temporal consistency can be added. X is the input video and Y is the resulting video output. (a) Frame-based model, (b) Temporal consistency is added in RecycleGAN bansal2018recycle and OfGANxu2020ofgan using optical flow or future frame prediction. (c) RT-GAN uses three consecutive output frames from generators are passed into a discriminator to provide temporal consistency. The first frame, $Y_{t-1}'$ is generated from a fully trained frame-based model. The other two frames are created by RT-GAN to be temporally consistent with $Y_{t-1}'$.
  • Figure 2: Comparisons of the results for RT-GAN (Ours) with stitched images from FoldIt, TempCycleGAN, and RecycleGAN on optical colonoscopy video dataset from ma2019real. Full results are found in the supplementary video.
  • Figure 3: Results for varying temporal weights ($\lambda$). The first row is the input and the second row shows $\lambda= 0.2$. As $\lambda$ decreases RT-GAN's is more faithful to FoldIt. The next row shows $\lambda=1$ where there is a balance between temporal and the frame losses. The last row shows $\lambda= 5$. Here the annotation shapes tend to remain consistent between frames. Full videos are found in the supplementary video.
  • Figure 4: Results for RT-GAN trained on CLTS-GAN. The top portion shows results on rendered mesh frames. CLTS-GAN's results change drastically over time. RT-GAN builds off CLTS-GAN to provide consistent specular and texture between frames. The bottom half shows results using OfGAN's input video, which embeds texture and specular information. CLTS-GAN adds more intricate specular reflections and textures, and RT-GAN inherits this property. OfGAN relies on the embedded texture and specular to produce its output. Full videos are in the supplementary video.