Tracking-Aware Deformation Field Estimation for Non-rigid 3D Reconstruction in Robotic Surgeries

TL;DR

The paper tackles the challenge of intraoperative tissue deformation awareness in robotic surgery by proposing Tracking-Aware Deformation Field (TADF), which couples 2D keypoint tracking with a neural implicit deformation field to recover accurate 3D tissue deformation. By sampling and tracking soft-tissue keypoints with a foundation model (CoTracker), interpolating 2D displacements to a dense heatmap, and guiding an implicit neural representation (SDF and radiance fields), TADF yields improved 3D deformation estimates and rendered reconstructions. Experimental results on EndoNeRF and SCARED datasets show that TADF achieves better deformation metrics and robust performance under noise compared to the EndoSurf baseline, with notable gains in PSNR, SSIM, and lower MSE/MaxSE. The approach provides a practical pathway toward safer surgical assistance by delivering deformation-aware 3D reconstructions and visualization, with future work aimed at real-time efficiency improvements using faster 3D rendering techniques.

Abstract

Minimally invasive procedures have been advanced rapidly by the robotic laparoscopic surgery. The latter greatly assists surgeons in sophisticated and precise operations with reduced invasiveness. Nevertheless, it is still safety critical to be aware of even the least tissue deformation during instrument-tissue interactions, especially in 3D space. To address this, recent works rely on NeRF to render 2D videos from different perspectives and eliminate occlusions. However, most of the methods fail to predict the accurate 3D shapes and associated deformation estimates robustly. Differently, we propose Tracking-Aware Deformation Field (TADF), a novel framework which reconstructs the 3D mesh along with the 3D tissue deformation simultaneously. It first tracks the key points of soft tissue by a foundation vision model, providing an accurate 2D deformation field. Then, the 2D deformation field is smoothly incorporated with a neural implicit reconstruction network to obtain tissue deformation in the 3D space. Finally, we experimentally demonstrate that the proposed method provides more accurate deformation estimation compared with other 3D neural reconstruction methods in two public datasets.

Figures (4)

• Figure 1: The proposed 3D Tracking-Aware Tissue Deformation Field (TADF) estimation framework. The colored circle represents each tracked key point.
• Figure 2: Overview of TADF. For input video frames, we first sample key points and track those sampled points with the foundation CoTracker model. Taking the key point displacement as an additional input, our designed neural deformation field could predict an accurate deformation of tissues.
• Figure 3: Intermediate Process Presentation for our Pipeline. Cutting means EndoNeRF-cutting dataset and Pulling means EndoNeRF-pulling dataset. Each subfigure represents a) reference - the input video frame, b) key point displacement - the sampled key points displacement tracked by foundation cotracker model, c) 2D deformation field - 2D visualization of deformation, d) 2D rendering - the rendering 2D videos after removing the surgical tools, e) reconstructed mesh - 3D reconstruction generated with the estimated deformation, and f) 3D deformation field - 3D visualization of deformation.
• Figure 4: Visualization of Deformation and 3D reconstruction. We take one key frame in EndoNeRF-cutting and EndoNeRF-pulling datasets to visualize estimated deformation and 3D reconstruction.