SeeBelow: Sub-dermal 3D Reconstruction of Tumors with Surgical Robotic Palpation and Tactile Exploration

TL;DR

This work addresses the lack of tactile sensing in robot-assisted MIS by developing a tactile navigation policy that combines GP-based search and contour-following impedance control to reconstruct sub-dermal tumor surfaces within a tri-layer tissue phantom. The method integrates scene registration, gravity-compensated force sensing, and online palpation, enabling high-fidelity 3D reconstructions of rigid tumor embeddings with fewer than 100 palpations. Key contributions include a 3D surface grid registration pipeline, BO-guided palpation point selection, and a contour-following strategy that traces tumor geometry to produce accurate tumor surface meshes. The approach demonstrates that contour-following palpations guided by Bayesian optimization outperform discrete probing in reconstruction quality, with potential for translation to RMIS platforms, though it currently omits friction modeling and fully soft-tumor scenarios.

Abstract

Surgical scene understanding in Robot-assisted Minimally Invasive Surgery (RMIS) is highly reliant on visual cues and lacks tactile perception. Force-modulated surgical palpation with tactile feedback is necessary for localization, geometry/depth estimation, and dexterous exploration of abnormal stiff inclusions in subsurface tissue layers. Prior works explored surface-level tissue abnormalities or single layered tissue-tumor embeddings with more than 300 palpations for dense 2D stiffness mapping. Our approach focuses on 3D reconstructions of sub-dermal tumor surface profiles in multi-layered tissue (skin-fat-muscle) using a visually-guided novel tactile navigation policy. A robotic palpation probe with tri-axial force sensing was leveraged for tactile exploration of the phantom. From a surface mesh of the surgical region initialized from a depth camera, the policy explores a surgeon's region of interest through palpation, sampled from bayesian optimization. Each palpation includes contour following using a contact-safe impedance controller to trace the sub-dermal tumor geometry, until the underlying tumor-tissue boundary is reached. Projections of these contour following palpation trajectories allows 3D reconstruction of the subdermal tumor surface profile in less than 100 palpations. Our approach generates high-fidelity 3D surface reconstructions of rigid tumor embeddings in tissue layers with isotropic elasticities, although soft tumor geometries are yet to be explored.

Figures (6)

• Figure 1: The experimental setup of the Robot (7DOF Franka-Panda) with custom built end effector housing a 3D load cell, an IMU sensor and a palpation tip probing a surgical tissue phantom. The phantom is further illustrated with an exploded view which incorporates - the Skin (Dermal Layer) 4mm thickness developed from silica gel, Fat (Adipose Layer) 15mm formed by a sponge material, and muscle (Myofascial Layer in Red) of stiffer foam material covering a rigid base, assembled to a support frame. Tumor geometries of different shapes, 3D printed with Markforged Onyx material, are placed in between muscle and fat layers to experiment 3D reconstruction of the tumor surface profile based on robotic palpation.
• Figure 2: The tactile exploration pipeline for tumor stiffness identification and generation of palpation trajectories. It comprises (a) Surgical Scene Registration by transforming the 3D surface meshes from depth camera point cloud into an interpolated 3D Surface Grid; (b) Optimal Search Planning to locate the next palpation points governed by Bayesian Optimization; (c) Finally the Online Palpation occurs at the searched palpation points from the skin surface to the tumor-muscle boundary using stiffness-estimates based contour following along the tumor contour, which generates the 3D palpation trajectories.
• Figure 3: Generation of interpolated 3D surface grid map from surgical scene. (a) 3D surface mesh created from depth camera's raw point cloud. (b) ROI markers specified by surgeon to create target area bounding box, whereas the dark shaded regions indicate the annotated tumor positions. (c) Generation of Interpolated 3D surface grid with cropped selected ROI (Red) overlayed on the grid cells enclosed by the bounding box.
• Figure 4: Representation of online palpation to detect presence of tumor using probing and palpation trajectory generation using contour following. (A) For Probing, Operational Space controller (OSC) is commanded to orient along the surface normal and indent at the optimal grid cell to detect presence of tumor. (B) The stiffness estimate is generated at the probed coordinate. If the force at maximum probe displacement exceeds a force threshold then tumor presence is confirmed at that coordinate. (C) If tumor presence is detected, then controller switches to contact-safe impedance control for tumor contour following. The probe travels along the contour of the tumor following the force threshold condition to reach the tumor-muscle boundary and terminates contour following. The trajectory way-points of the contour following yields the palpation trajectory further used for tumor surface 3D reconstruction.
• Figure 5: 3D visualization of generated palpation trajectories from contour-following of the tumor surface, which takes the shape of the tumor's surface profile. The oscillations with higher amplitude is observed at the edge of the tumor i.e. tumor-muscle boundary. The cropped surface mesh of the phantom is displayed above the tumor surface