Tracking and Mapping in Medical Computer Vision: A Review

TL;DR

This survey provides a comprehensive, algorithm-focused review of camera-based tracking and mapping in medical computer vision, with emphasis on deformable environments encountered in endoscopy, bronchoscopy, and related diagnostic and surgical contexts. It compiles a large literature base (516 papers) and organizes it into medical specialties, datasets, and algorithmic families (features, mosaicking, depth, tissue tracking, SfM/NRSfM/SfT, SLAM), while discussing evaluation metrics and practical challenges. The authors highlight the need for deformable models, richer real-tissue datasets, improved evaluation protocols, and the integration of neural rendering and learned priors to bridge gaps between rigid and nonrigid processing in clinical settings. They conclude with directions and questions focused on adaptive map management, uncertainty quantification, and scalable, real-time deployment to advance clinical impact.

Abstract

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Figures (17)

• Figure 1: Histogram of publications after final filtering. This search was performed on July 15th, 2023, and thus not all publications from 2023 are necessarily included in this study. Neural denotes publications which have a title or abstract containing: CNN, GNN, or neural network.
• Figure 2: A Sankey diagram of the paper filtering process.
• Figure 3: A body model with medical specialties that use tracking and mapping overlaid. Images are adapted from: (a). basergaEfficacyAutologousFat2020, (b). richaRobust3DVisual2011, (c). maRNNSLAMReconstructing3D2021, (d). soperSurfaceMosaicsBladder2012, (e). marmolDenseArthroSLAMDenseIntraArticular2019, (f). banoPlacentalVesselguidedHybrid2023, (g). schmidtSENDDSparseEfficient2023, (h). borrego-carazoBronchoPoseAnalysisData2023, (i). burschkaScaleinvariantRegistrationMonocular2005, (j). jiCorticalSurfaceShift2014. Permissions: (a, f). licensed under CC BY 4.0. (b, c, h, i, j). reprinted with permission from Elsevier. (d, e). reprinted with permission from IEEE. (g). reprinted with permission from authors. The 3D body model is generated and used with permission from BioDigital.
• Figure 4: Histogram of publicly available datasets usable for camera-based tracking and mapping in MCV from datasets mentioned in Table \ref{['tab:datasets']}.
• Figure 5: Dataset simulation framework for multi-camera systems to evaluate image stitching algorithms. From guyQualitativeComparisonImage2022 licensed under CC BY 4.0