Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training

Nisarg A. Shah, Wele Gedara Chaminda Bandara, Shameema Skider, S. Swaroop Vedula, Vishal M. Patel

TL;DR

Cataract surgery video analysis suffers from limited labeled data for downstream tasks such as step recognition. The authors propose CSMAE, a MAE-based pretraining framework that uses a Token Selection Network to sample spatiotemporally informative tokens, enabling very high masking ratios and efficient learning from unlabeled videos; the model employs a ViT-based encoder and a lightweight decoder, trained with a reconstruction objective and a token-selection objective learned through reinforcement-like updates. On D99 and Cataract-101, CSMAE outperforms prior self-supervised and adapter-based methods, with notable gains in low-label regimes (approximately 8% uplift at 10% labels) and strong transferability (3–5% improvements in Jaccard and recall). This work demonstrates a robust, data-efficient pretraining backbone for surgical video analysis with potential applicability across diverse procedures and datasets.

Abstract

Automated analysis of surgical videos is crucial for improving surgical training, workflow optimization, and postoperative assessment. We introduce a CSMAE, Masked Autoencoder (MAE)-based pretraining approach, specifically developed for Cataract Surgery video analysis, where instead of randomly selecting tokens for masking, they are selected based on the spatiotemporal importance of the token. We created a large dataset of cataract surgery videos to improve the model's learning efficiency and expand its robustness in low-data regimes. Our pre-trained model can be easily adapted for specific downstream tasks via fine-tuning, serving as a robust backbone for further analysis. Through rigorous testing on a downstream step-recognition task on two Cataract Surgery video datasets, D99 and Cataract-101, our approach surpasses current state-of-the-art self-supervised pretraining and adapter-based transfer learning methods by a significant margin. This advancement not only demonstrates the potential of our MAE-based pretraining in the field of surgical video analysis but also sets a new benchmark for future research.

$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training

TL;DR

Cataract surgery video analysis suffers from limited labeled data for downstream tasks such as step recognition. The authors propose CSMAE, a MAE-based pretraining framework that uses a Token Selection Network to sample spatiotemporally informative tokens, enabling very high masking ratios and efficient learning from unlabeled videos; the model employs a ViT-based encoder and a lightweight decoder, trained with a reconstruction objective and a token-selection objective learned through reinforcement-like updates. On D99 and Cataract-101, CSMAE outperforms prior self-supervised and adapter-based methods, with notable gains in low-label regimes (approximately 8% uplift at 10% labels) and strong transferability (3–5% improvements in Jaccard and recall). This work demonstrates a robust, data-efficient pretraining backbone for surgical video analysis with potential applicability across diverse procedures and datasets.

Abstract

Automated analysis of surgical videos is crucial for improving surgical training, workflow optimization, and postoperative assessment. We introduce a CSMAE, Masked Autoencoder (MAE)-based pretraining approach, specifically developed for Cataract Surgery video analysis, where instead of randomly selecting tokens for masking, they are selected based on the spatiotemporal importance of the token. We created a large dataset of cataract surgery videos to improve the model's learning efficiency and expand its robustness in low-data regimes. Our pre-trained model can be easily adapted for specific downstream tasks via fine-tuning, serving as a robust backbone for further analysis. Through rigorous testing on a downstream step-recognition task on two Cataract Surgery video datasets, D99 and Cataract-101, our approach surpasses current state-of-the-art self-supervised pretraining and adapter-based transfer learning methods by a significant margin. This advancement not only demonstrates the potential of our MAE-based pretraining in the field of surgical video analysis but also sets a new benchmark for future research.

Paper Structure

This paper contains 6 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. The $\mathsf{CSMAE~}$ model
  3. Training $\mathsf{CSMAE~}$
  4. Experiments and Results
  5. Conclusion
  6. Acknowledgments

Figures (2)

  • Figure 1: The $\mathsf{CSMAE~}$ model, developed on an encoder-decoder framework, consists of four principal components: a Tokenizer, Token Selection Network, an Encoder, and a Lightweight Decoder. The Tokenizer converts raw video data into a token-based feature representation. Utilizing transformer architecture, the Token Selection Network predicts the probability distribution for each token, choosing those with significant spatiotemporal information to feed into the Encoder. This Encoder, which employs the ViT network, captures the feature representation of these selected tokens. These representations are then combined with learnable feature representations of masked tokens, aiming to reconstruct the masked feature representation accurately.
  • Figure 2: Qualitative Comparison of Reconstructed Images after Pre-training Experiment. (a) indicates Original Images, (b) indicates Mask and Reconstructed Image from VideoMAEtong2022videomae and (c) indicates Selected Masks from TSN and Reconstructed Image from $\mathsf{CSMAE~}$.