Visual Attention Based Cognitive Human-Robot Collaboration for Pedicle Screw Placement in Robot-Assisted Orthopedic Surgery

TL;DR

This work addresses the safety and workload challenges in robot-assisted pedicle screw placement by introducing a cognitive human-robot collaboration framework that keeps the surgeon in the loop. It integrates an AR-Haptic interface for intuitive command and feedback, an eye-tracking based surgeon attention model, and a shared-control scheme that adaptively allocates autonomy via an attention-driven weight $w$. The approach combines a geometric, affine-aligned positioning strategy and impedance-based haptic control to synchronize robot and human contributions, with $K(w)$ shaping stiffness along drilling axes and $f_{fdbk}$ modulating sensory feedback. Experimental validation on a UR5-based platform with a bone drill, haptic input, and AR visualization demonstrates improved safety and ergonomics over full-robot or full-human control, highlighting potential clinical impact and design principles for cognitive shared control in safety-critical surgical robotics.

Abstract

Current orthopedic robotic systems largely focus on navigation, aiding surgeons in positioning a guiding tube but still requiring manual drilling and screw placement. The automation of this task not only demands high precision and safety due to the intricate physical interactions between the surgical tool and bone but also poses significant risks when executed without adequate human oversight. As it involves continuous physical interaction, the robot should collaborate with the surgeon, understand the human intent, and always include the surgeon in the loop. To achieve this, this paper proposes a new cognitive human-robot collaboration framework, including the intuitive AR-haptic human-robot interface, the visual-attention-based surgeon model, and the shared interaction control scheme for the robot. User studies on a robotic platform for orthopedic surgery are presented to illustrate the performance of the proposed method. The results demonstrate that the proposed human-robot collaboration framework outperforms full robot and full human control in terms of safety and ergonomics.

Figures (7)

• Figure 1: The framework of the proposed human--robot interface. AR scene displays surgical information, while the haptic device enables intuitive operation. Their positions are coupled to enhance hand-eye coordination.
• Figure 2: Gaze points are obtained via ray-casting on the AR scene or the spatial mesh. The attention level is estimated by the proportion of fixation duration on surgery-related objects within a time window.
• Figure 3: Geometry relationship between the robot and the haptic device.
• Figure 4: Experimental Setup. (a) The proposed AR-haptic interface captured by HoloLens 2. (b) The end-effector design of the robot.
• Figure 5: Snapshots from a representative experiment. The participant's gaze point is indicated by a green dot. (a) The participant started drilling. (b) Drilling in progress. (c) Distraction phase: participant shifted attention to solving mental arithmetic problems. (d) Drilling task completed.