Training-free Temporal Object Tracking in Surgical Videos

TL;DR

A novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos, targeting localisation and tracking of critical anatomical structures and instruments using text-to-image diffusion models, offering a promising avenue for accurate and cost-effective temporal object tracking in minimally invasive surgery videos.

Abstract

Purpose: In this paper, we present a novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos, targeting localisation and tracking of critical anatomical structures and instruments. Our method addresses the challenges of costly pixel-level annotations and label inconsistencies inherent in existing datasets. Methods: Leveraging the inherent object localisation capabilities of pre-trained text-to-image diffusion models, we extract representative features from surgical frames without any training or fine-tuning. Our tracking framework uses these features, along with cross-frame interactions via an affinity matrix inspired by query-key-value attention, to ensure temporal continuity in the tracking process. Results: Through a pilot study, we first demonstrate that diffusion features exhibit superior object localisation and consistent semantics across different decoder levels and temporal frames. Later, we perform extensive experiments to validate the effectiveness of our approach, showcasing its superiority over competitors for the task of temporal object tracking. Specifically, we achieve a per-pixel classification accuracy of 79.19%, mean Jaccard Score of 56.20%, and mean F-Score of 79.48% on the publicly available CholeSeg8K dataset. Conclusion: Our work not only introduces a novel application of text-to-image diffusion models but also contributes to advancing the field of surgical video analysis, offering a promising avenue for accurate and cost-effective temporal object tracking in minimally invasive surgery videos.

Figures (7)

• Figure 1: Our method uses pre-trained text-to-image diffusion model rombach2022high to extract features from surgical frames (Sec. \ref{['sec:diff_feat']}). Our tracking module (Sec. \ref{['sec:seg']}) uses these features with cross-frame interactions via an affinity matrix, to predict masks in the temporal direction for the entire clip.
• Figure 2: PCA rendering of SD's $\mathcal{U}_{\mathbf{u}}^{n}$ features from different levels ($n=$$\{1,2,3,4\}$) and time-frames. Notably, same anatomy/instrument is represented by same colour across temporal direction.
• Figure 3: Proposed diffusion feature extraction pipeline. Given a surgical frame $\mathbf{x}_0$, it is passed through the encoder $\mathcal{E}(\cdot)$ to generate a latent representation $\mathbf{z}_0$, followed by forward diffusion to produce a noisy latent $\mathbf{z}_t$. The null prompt $\mathcal{T}(\mathbf{c})$ and noisy latent representation $\mathbf{z}_t$ are then passed through the UNet $\mathcal{U}(\cdot)$ to extract features from its internal decoders (Sec. \ref{['sec:diff_feat']}).
• Figure 4: Qualitative comparison on CholecSeg8K hong2020cholecseg8k. Notably, unlike the baseline competitors, the proposed method precisely tracks both smaller and larger anatomies across the temporal direction.
• Figure 5: Qualitative comparison on CholecSeg8K hong2020cholecseg8k. Noticeably, the proposed method surpasses competitors, maintaining accuracy even during rapid temporal movements of instruments and anatomis.