Probabilistic Task Parameterization of Tool-Tissue Interaction via Sparse Landmarks Tracking in Robotic Surgery
Yiting Wang, Yunxin Fan, Fei Liu
TL;DR
The paper tackles robust tool–tissue interaction modeling in robotic minimally invasive surgery under nonrigid tissue deformation by fusing sparse landmark tracking with a time-conditioned TP-GMM. Tissue frames are dynamically generated from clustered landmarks via PCA, and tool poses are expressed relative to these local frames, enabling uncertainty-aware, context-sensitive predictions from endoscopic video. By integrating data-driven observations with surgical priors through TP-GMM, the approach delivers deformation-aware trajectory predictions and interpretable visualizations, with potential applications in Learning from Demonstration and Imitation Learning. Experiments on the EndoNeRF dataset reveal strong training performance but reveal generalization gaps, guiding future work on data augmentation and regularization.
Abstract
Accurate modeling of tool-tissue interactions in robotic surgery requires precise tracking of deformable tissues and integration of surgical domain knowledge. Traditional methods rely on labor-intensive annotations or rigid assumptions, limiting flexibility. We propose a framework combining sparse keypoint tracking and probabilistic modeling that propagates expert-annotated landmarks across endoscopic frames, even with large tissue deformations. Clustered tissue keypoints enable dynamic local transformation construction via PCA, and tool poses, tracked similarly, are expressed relative to these frames. Embedding these into a Task-Parameterized Gaussian Mixture Model (TP-GMM) integrates data-driven observations with labeled clinical expertise, effectively predicting relative tool-tissue poses and enhancing visual understanding of robotic surgical motions directly from video data.