Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model

TL;DR

This work tackles data imbalance in surgical video datasets by generating synthetic videos for rare actions using SurgiFlowVid, a diffusion-based framework. It introduces a dual-prediction diffusion U-Net that jointly models RGB frames and optical flow, plus a sparse visual encoder to enable controllable generation with sparse annotations. Built on a two-stage training scheme adapted from SurV-Gen, SurgiFlowVid delivers temporally coherent, diverse samples even when conditioning signals are sparse. Across datasets and tasks—action recognition, tool presence detection, and laparoscope motion prediction—it yields consistent 10–20% gains over strong baselines, highlighting its practical potential for robust surgical video understanding.

Abstract

Surgical video datasets are essential for scene understanding, enabling procedural modeling and intra-operative support. However, these datasets are often heavily imbalanced, with rare actions and tools under-represented, which limits the robustness of downstream models. We address this challenge with $SurgiFlowVid$, a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes. Our approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples. In addition, a sparse visual encoder conditions the generation process on lightweight signals (e.g., sparse segmentation masks or RGB frames), enabling controllability without dense annotations. We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction. Synthetic data generated by our method yields consistent gains of 10-20% over competitive baselines, establishing $SurgiFlowVid$ as a promising strategy to mitigate data imbalance and advance surgical video understanding methods.

Figures (16)

• Figure 1: Data challenge in the surgical domain. During a laparoscopic procedure, the surgeon operates via the endoscopic video feed (video on the monitor). ML models can leverage these videos for providing guidance through surgical scene understanding. However, the datasets are skewed as shown in the bar plots. We aim to mitigate data imbalance with synthetic samples. The right plot shows improvements from adding samples generated from our approach (SurgiFlowVid).
• Figure 2: SurgiFlowVid approach.The dual-prediction diffusion U-Net module reconstructs both RGB and optical flow frames from noised inputs to capture spatio-temporal dynamics from limited data. Sparse visual encoder is trained with segmentation masks (if available) or RGB frames for conditional generation; optical flow is used only during training.
• Figure 3: Qualitative results of the action "tie the suture." Purple boxes denote the sparse RGB conditioning frames. Suprious tools are generated in SurV-Gen (white box, row $1$), while SparseCtrl alters tool types compared to the conditioning frames (white box, row $2$), reflecting limited spatial consistency. SurgiFlowVid preserves both spatial and temporal structure, with consistent tools maintained across generated frames (yellow boxes, row $3$).
• Figure 4: Laparoscope motion prediction in online (left) and offline (right) fashion on the AutoLapro dataset.
• Figure 5: Tool types match the sparse seg. frames, but their position shifts, causing a failure case.