Haptics-Enabled Forceps with Multi-Modal Force Sensing: Towards Task-Autonomous Surgery

TL;DR

This work tackles the absence of tool-side force sensing in micro-sized MIS forceps by introducing a vision-based, multi-modal sensing module that works in tandem with a proximal single-axis sensor to deliver haptic information. The approach combines a cylindrical sensing module (flexure-camera-target), a robust pose-estimation and marker-registration pipeline, and explicit force estimations for pushing/pulling ($F_p$) and grasping ($F_g$) modes, all validated through phantom and ex vivo experiments. Key contributions include (i) a compact, integrable sensing module design, (ii) a pose-estimation and registration algorithm that tolerates occlusions, (iii) closed-form force estimation for Mode-I and Mode-II, calibrated via a stiffness matrix $\mathbf{K}_s$, and (iv) demonstration of autonomous ex vivo tissue grasping using a UR-5 robot. The results indicate accurate multi-modal force sensing and reveal the method’s potential to enable autonomous tissue manipulation in MIS, with implications for safety, precision, and repeatability in robotic surgery.

Abstract

Many robotic surgical systems have been developed with micro-sized forceps for tissue manipulation. However, these systems often lack force sensing at the tool side and the manipulation forces are roughly estimated and controlled relying on the surgeon's visual perception. To address this challenge, we present a vision-based module to enable the micro-sized forceps' multi-modal force sensing. A miniature sensing module adaptive to common micro-sized forceps is proposed, consisting of a flexure, a camera, and a customised target. The deformation of the flexure is obtained by the camera estimating the pose variation of the top-mounted target. Then, the external force applied to the sensing module is calculated using the flexure's displacement and stiffness matrix. Integrating the sensing module into the forceps, in conjunction with a single-axial force sensor at the proximal end, we equip the forceps with haptic sensing capabilities. Mathematical equations are derived to estimate the multi-modal force sensing of the haptics-enabled forceps, including pushing/pulling forces (Mode-I) and grasping forces (Mode-II). A series of experiments on phantoms and ex vivo tissues are conducted to verify the feasibility of the proposed design and method. Results indicate that the haptics-enabled forceps can achieve multi-modal force estimation effectively and potentially realize autonomous robotic tissue grasping procedures with controlled forces. A video demonstrating the experiments can be found at https://youtu.be/pi9bqSkwCFQ.

Figures (9)

• Figure 1: Typical tissue manipulations using a pair of forceps in robotic thyroid tumor treatment, an example scenario of robotic surgery, and the critically required force information. (a) A pair of forceps touches the targeted tissue while measuring the pushing forces. (b) The forceps grasp the targeted tissue while measuring and controlling the grasping force. (c) The forceps pull the grasped tissue while measuring and controlling the pulling force. (d) The micro-level actuator used for driving the forceps, where a commercial single-axis force sensor measures the driving force applied to the forceps' driving cable. The upper and lower linear drivers are responsible for grasping and pushing/pulling, respectively. The lower driver also compensates for the motion introduced by the flexure's deformation when the upper driver grasps tissue. (e) The haptics-enabled biopsy forceps, where the forceps' base is concentrically installed to the vison-based sensing module's head (2). Two LEDs (1) are mounted in the sensing module's head (2) to provide light source in our prototype, which comes through the target's (3) holes and is captured by the camera (5) mounted on the other end of the flexure (4). The upper-right inset shows the installed target and the holes used as markers. The cylindrical base (6) provides a mount to connect to instruments, which is a stainless tube (8) in our prototype. Concentric to the module, a channel is reserved for the passage of the forceps' driving cable (7). The lower-right inset shows a prototype with a 4mm diameter and 22mm length. Plot (e) also shows the camera frame {$C$}, the target frames of the initial {$M_0$} and current {$M_t$} states, and the forces applied to the forceps when they push or pull the tissue. $\mathbf{f}_d$ is the driving force on the cable and is measured by the proximal commercial force sensor. $\mathbf{f}^{\prime}_d$ is the driving force transmitted to the forceps' jaws through the central cable. $\mathbf{f}_s$ is the supporting force from the sensing module, and $\mathbf{f}_p$ is the pushing/pulling force from the tissue.
• Figure 2: (a) The projection relationship between the $i$-th marker ($^{M}{\mathbf{p}}_{i}$) in the target frame {M} and its corresponding pixel position ($^{I}{\mathbf{p}}_{i}$) in image frame {I}. (a) also shows the original index of markers. (b) The directly captured image at the initial state. (c) The blob detected result at the initial state based on the filtered image. These detected marker centers are returned for pose estimation. (d) The blob detection result at a random pose, where the indexes of detected markers differ, and the points labeled with 3, 6, and 11 in the initial state are lost. (e) The registration result of (e), where markers were registered to their original indexes.
• Figure 3: (a) The sectional view of the haptics-enabled forceps when the two jaws are closed. (b) shows a state when the two jaws are open at $\theta$. $t_d$ is the movement of the driving cable, while $t_s$ is the transformation of the sensing module's head estimated by the camera. (c) The forces applied to the forceps when they grasp and pull a tissue include driving force $F_d$ from the cable, supporting force $F_s$ from the sensing module, contact force $F_g$ (also called grasping force), and friction force $F_f$ from the tissue. The middle inset sketched the forces applied to the tissue, where $F_p$ is the pulling force from the tissue body, $F^\prime_g = F_g$ and $F^\prime_f = F_f$ are from the forceps. The right inset shows the forces that generate the momentum of one jaw about joint $j_2$. $F_d$ is measured by the proximal force sensor, while the driving distance $t_d$ is measured by the upper driver's encoder of the micro-level actuator.
• Figure 4: (a) The experimental setup for evaluating pose estimation. The inset shows the configuration, where the EM tracking sensor was installed concentrically to the target. (b) The sensor's head was moved towards $x$, $y$ and $z$ directions during the experiments, and $t_z$ denotes the transformation along $z$ axis. (c) The orientation comparison between nine pairs of points that were estimated by the EM tracking system ($\star$) and camera ($\ast$). (d) The position comparison between the continued EM tracking and camera estimation results.
• Figure 5: (a) The experimental setup for stiffness matrix calibration. The forceps were pulled by a cable connected with changeable weights. The orientation module was used to adjust the pull direction of the forceps. (b) shows the setup for calibration data collection. (c) shows the setup for verification data collection, where the orientation module was rotated 45$^\circ$ relative to that for calibration. $v_1$, $v_2$, $v_3$, and $v_4$ indicate the pulling directions for verification in $x$-$y$ panel. (d) and (e) show the verification results in $x$-$y$ panel and $z$-axial direction, respectively. $F_w$ and $F_e$ denote the force generated by the weight and the estimated result.