On the Fly Robotic-Assisted Medical Instrument Planning and Execution Using Mixed Reality

TL;DR

A novel framework using mixed reality technologies to ease the use of RAMS achieves real-time planning and execution of medical instruments by providing 3D anatomical image overlay, human-robot collision detection, and robot programming interface.

Abstract

Robotic-assisted medical systems (RAMS) have gained significant attention for their advantages in alleviating surgeons' fatigue and improving patients' outcomes. These systems comprise a range of human-computer interactions, including medical scene monitoring, anatomical target planning, and robot manipulation. However, despite its versatility and effectiveness, RAMS demands expertise in robotics, leading to a high learning cost for the operator. In this work, we introduce a novel framework using mixed reality technologies to ease the use of RAMS. The proposed framework achieves real-time planning and execution of medical instruments by providing 3D anatomical image overlay, human-robot collision detection, and robot programming interface. These features, integrated with an easy-to-use calibration method for head-mounted display, improve the effectiveness of human-robot interactions. To assess the feasibility of the framework, two medical applications are presented in this work: 1) coil placement during transcranial magnetic stimulation and 2) drill and injector device positioning during femoroplasty. Results from these use cases demonstrate its potential to extend to a wider range of medical scenarios.

Figures (7)

• Figure 1: System diagram. Arrows indicate the data flow directions within the system. The system consists of three main modules: The instrument placement planner, collision object converter, and robot programming interface. The instrument planner (top left block) processes user input and visualizes overlaid medical images such as MRI, as well as previews the instrument placement. The collision object converter (bottom left block), built on the motion tracking and spatial localization functionalities of MR-HMD, maps the operator's motion to a customized avatar and then converts the avatar to collision objects for robot trajectory planning. The robot programming interface (right block) is responsible for trajectory planning, preview of the planned trajectory, and waypoint adjustment. Once the operator confirms the trajectory, the robot will automatically execute the instrument placement.
• Figure 2: Kinematics of the system using robot-assisted TMS as the use case. The solid arrows denote the transformations directly obtained by the external tracking system, HMD, or calibration/registration, while the dashed arrows denote the derived transformations. Note: colored RGB axes and $\{\cdot\}$ represent coordinate systems.
• Figure 3: The communication network of the system. Four main components — HoloLens 2, Unity, ROS, and Hardware — are connected using various methods. Each component encapsulates sub-elements divided by functionality. Ethernet is used for robot and sensor connections, and Transmission Control Protocol/Internet Protocol (TCP/IP) and User Datagram Protocol (UDP) are used for wireless communication.
• Figure 4: Kinematics of the virtual-to-real calibration and its evaluation method. (a) The kinematic chain of the calibration method. Two optical markers are used: one is fixed on the HoloLens 2, and the other is free to move and serves as a reference. The transformation from HoloLens 2 to the reference marker is measured through the STTAR system martin2023sttar. (b) The kinematic chain of the evaluation method in which the operator aligns the virtual marker $VO_{ref}$ to the real one $O_{ref}$. Note: the notations follow the same convention as outlined in Fig. \ref{['fig:system kinamatics']}
• Figure 5: Experimental setup. (a) HoloLens 2 with optical markers attached. (b) An avatar of the operator, with collision objects (simple geometric shapes) attached. (c) User interface. (d) TMS scene consisting of a TMS coil and a phantom head. (e) A cortical location pointed by the clicker. (f) Trajectory planning of the robot in robot-assisted TMS. (g) Femoroplasty scene with a drill and injector device, and a phantom femur. (h) Collision objects in MoveIt!. (i) Trajectory planning of the robot in robotic-assisted femoroplasty.