Multimodal Feedback for Handheld Tool Guidance: Combining Wrist-Based Haptics with Augmented Reality

TL;DR

This work investigates enhancing handheld surgical tool guidance by integrating a wrist-worn vibrotactile haptic device with optical see-through AR. A formative study established a fixed wrist-up reference frame and identified key tool maneuvers to encode into five haptic patterns, followed by a cue-identification study (N=21) and a surgical targeting task in a knee phantom (N=27). Results show high cue recognizability, particularly for full-actuator cues, and that AR+Haptics yields the best spatial precision (5.8 mm) and usability (SUS 88.1), albeit with longer task times compared to AR alone. The findings support multimodal guidance as a promising approach for high-precision, visually challenging tasks, and offer concrete design guidance for integrating wrist-based haptics with AR in surgical contexts.

Abstract

We investigate how vibrotactile wrist feedback can enhance spatial guidance for handheld tool movement in optical see-through augmented reality (AR). While AR overlays are widely used to support surgical tasks, visual occlusion, lighting conditions, and interface ambiguity can compromise precision and confidence. To address these challenges, we designed a multimodal system combining AR visuals with a custom wrist-worn haptic device delivering directional and state-based cues. A formative study with experienced surgeons and residents identified key tool maneuvers and preferences for reference mappings, guiding our cue design. In a cue identification experiment (N=21), participants accurately recognized five vibration patterns under visual load, with higher recognition for full-actuator states than spatial direction cues. In a guidance task (N=27), participants using both AR and haptics achieved significantly higher spatial precision (5.8 mm) and usability (SUS = 88.1) than those using either modality alone, despite having modest increases in task time. Participants reported that haptic cues provided reassuring confirmation and reduced cognitive effort during alignment. Our results highlight the promise of integrating wrist-based haptics into AR systems for high-precision, visually complex tasks such as surgical guidance. We discuss design implications for multimodal interfaces supporting confident, efficient tool manipulation.

Figures (8)

• Figure 1: [id=+]The custom haptic wristband used in the formative study embedded vibrotactile motors for direct skin contact. Its flexible design conformed to different wrist sizes. Three motors defined an “up” direction relative to gravity, with one selected based on initial tool orientation, while a fourth motor defined a wrist-relative “up” direction that remained constant across participants.
• Figure 2: [id=+]Reference orientations and common tool movements from the formative study. (a) compares fixed versus adaptive reference frame mappings. (b–d) depict tool maneuvers, directional movements, and out-of-plane motions identified by users, which informed the vibration pattern design and subsequent experiments.
• Figure 3: Haptic system design. A wristband with 12 individually controlled vibrotactile motors was driven by an Arduino microcontroller, which communicated with the HoloLens via serial connection. Tool motion was tracked using the HoloLens depth camera, and vibration amplitude and frequency were measured with an embedded accelerometer. [id=+]Amplitude is in 9.81 m/s² (1 gravitational acceleration).
• Figure 4: Vibrotactile patterns encoding five tool actions. We designed five haptic cue sequences to guide tool movement during surgical tasks: (a) planar directional movement, or Move to, (b) Move up, (c) Move down, (d) Pause, and (e) Arrived. Continuous movement cues—planar and out-of-plane—varied in amplitude, frequency, actuator count, and inter-pulse intervals. Out-of-plane cues were triggered when tool position deviated more than ±5 mm from the defined movement plane, activating five top-side (up) or bottom-side (down) actuators. Upon re-entering the valid plane, a discrete pause signal was issued using two bursts from all actuators. When the tool reached within 10 mm of the target center, a 3-second full-actuator signal confirmed successful arrival. Patterns were designed to be easily distinguishable and grounded in formative studies on surgical tool maneuvers.
• Figure 5: Mean recognition rates for the five haptic cue categories (Move to, Move up, Move down, Pause, and Arrived), averaged across all participants. Darker colors indicate higher percentiles.