Efficient Surgical Tool Recognition via HMM-Stabilized Deep Learning
Haifeng Wang, Hao Xu, Jun Wang, Jian Zhou, Ke Deng
TL;DR
This work tackles surgical tool recognition and phase detection in videos by arguing that surgical workflows exhibit a compact temporal structure amenable to a Hidden Markov Model (HMM). It introduces an HMM-stabilized deep learning framework that couples a phase Markov chain with per-phase tool-presence chains, leveraging DL predictions through probabilistic emissions and enabling stable, interpretable inference via EM and Viterbi methods. Semi-supervised learning and an annotation-efficient Lobec100 data-collection strategy demonstrate that the proposed approach achieves competitive accuracy while reducing training cost and enabling flexible use of partially labeled data. The results underscore the value of integrating statistical learning with deep learning for structured unstructured data, offering improved robustness, transparency, and practicality for surgical video analysis.
Abstract
Recognizing various surgical tools, actions and phases from surgery videos is an important problem in computer vision with exciting clinical applications. Existing deep-learning-based methods for this problem either process each surgical video as a series of independent images without considering their dependence, or rely on complicated deep learning models to count for dependence of video frames. In this study, we revealed from exploratory data analysis that surgical videos enjoy relatively simple semantic structure, where the presence of surgical phases and tools can be well modeled by a compact hidden Markov model (HMM). Based on this observation, we propose an HMM-stabilized deep learning method for tool presence detection. A wide range of experiments confirm that the proposed approaches achieve better performance with lower training and running costs, and support more flexible ways to construct and utilize training data in scenarios where not all surgery videos of interest are extensively labelled. These results suggest that popular deep learning approaches with over-complicated model structures may suffer from inefficient utilization of data, and integrating ingredients of deep learning and statistical learning wisely may lead to more powerful algorithms that enjoy competitive performance, transparent interpretation and convenient model training simultaneously.