Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models

Danush Kumar Venkatesh, Isabel Funke, Micha Pfeiffer, Fiona Kolbinger, Hanna Maria Schmeiser, Juergen Weitz, Marius Distler, Stefanie Speidel

TL;DR

This work tackles data imbalance in surgical video datasets by introducing SurV-Gen, a two-stage diffusion-based framework that first learns spatial content via fine-tuning a pre-trained Stable Diffusion and then models motion with temporal transformers in a frozen-spatial-weights setup. Conditioning on text prompts and class labels enables targeted generation of under-represented classes, while a rejection sampling scheme curates high-quality synthetic samples for downstream tasks. Evaluations on surgical action recognition and staple line bleeding detection show that incorporating RS-filtered synthetic videos yields substantial improvements over real data alone and strong baselines. The approach demonstrates the practical value of diffusion-generated videos for augmenting scarce, imbalanced surgical data and is released as open-source for broad adoption.

Abstract

Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at https://gitlab.com/nct_tso_public/surgvgen.

Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models

TL;DR

This work tackles data imbalance in surgical video datasets by introducing SurV-Gen, a two-stage diffusion-based framework that first learns spatial content via fine-tuning a pre-trained Stable Diffusion and then models motion with temporal transformers in a frozen-spatial-weights setup. Conditioning on text prompts and class labels enables targeted generation of under-represented classes, while a rejection sampling scheme curates high-quality synthetic samples for downstream tasks. Evaluations on surgical action recognition and staple line bleeding detection show that incorporating RS-filtered synthetic videos yields substantial improvements over real data alone and strong baselines. The approach demonstrates the practical value of diffusion-generated videos for augmenting scarce, imbalanced surgical data and is released as open-source for broad adoption.

Abstract

Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at https://gitlab.com/nct_tso_public/surgvgen.
Paper Structure (12 sections, 3 equations, 3 figures, 3 tables)

This paper contains 12 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Stage 1: Fine-tuning Stable Diffusion
  4. Stage 2: Learning motion dynamics
  5. Selection of generated data
  6. Evaluation
  7. Experiments
  8. Downstream tasks
  9. Results and discussion
  10. Conclusion
  11. Acknowledgments.
  12. Disclosure of Interests.

Figures (3)

  • Figure 1: Data imbalance among classes in surgical datasets. Left: Frequency of different actions (G0-G7) in the SAR-RARP50 psychogyios2023sar dataset. Right: Occurrence of staple line bleeding in distal pancreatectomy.
  • Figure 2: Overview of the SurV-Gen method for surgical video generation.
  • Figure 3: Qualitative comparison of generated video frames. In our approach (SurV-Gen), the scissors are clearly visible during suture cutting (row 3), whereas in other methods, the scissors appear only partially. Similarly, during the needle return, the tools are consistently generated (row 6).