Augmented reality for upper limb rehabilitation: real-time kinematic feedback with HoloLens 2

TL;DR

The paper addresses the need for precise, real-time kinematic feedback in upper-limb rehabilitation for neurological disorders by leveraging HoloLens 2 to overlay holographic 3D trajectories and provide color-coded accuracy feedback during training. It implements an AR system integrated with the AGREE upper-limb exoskeleton, using calibration, trajectory manipulation, and execution phases to deliver real-time error cues. Empirical results from 15 healthy subjects show improved kinematic accuracy with AR feedback across four tasks, while 12 clinicians report acceptable usability and meaningful acceptance (SUS 67.7; TAM indicates willingness to use). The work demonstrates AR-based 3D guidance as a promising adjunct to robotic rehabilitation, with potential to enhance therapist feedback and patient outcomes, and outlines clear directions for clinical validation in neurological populations.

Abstract

Exoskeletons for rehabilitation can help enhance motor recovery in individuals suffering from neurological disorders. Precision in movement execution, especially in arm rehabilitation, is crucial to prevent maladaptive plasticity. However, current exoskeletons, while providing arm support, often lack the necessary 3D feedback capabilities to show how well rehabilitation exercises are being performed. This reduces therapist acceptance and patients' performance. Augmented Reality technologies offer promising solutions for feedback and gaming systems in rehabilitation. In this work, we leverage HoloLens 2 with its advanced hand-tracking system to develop an application for personalized rehabilitation. Our application generates custom holographic trajectories based on existing databases or therapists' demonstrations, represented as 3D tunnels. Such trajectories can be superimposed on the real training environment. They serve as a guide to the users and, thanks to colour-coded real-time feedback, indicate their performance. To assess the efficacy of the application in improving kinematic precision, we tested it with 15 healthy subjects. Comparing user tracking capabilities with and without the use of our feedback system in executing 4 different exercises, we observed significant differences, demonstrating that our application leads to improved kinematic performance. 12 clinicians tested our system and positively evaluated its usability (System Usability Scale score of 67.7) and acceptability (4.4 out of 5 in the 'Willingness to Use' category in the relative Technology Acceptance Model). The results from the tests on healthy participants and the feedback from clinicians encourage further exploration of our framework, to verify its potential in supporting arm rehabilitation for individuals with neurological disorders.

Figures (9)

• Figure 1: HoloLens 2 Headset MicrosoftHoloLens2024
• Figure 2: Representation of the calibration procedure: (i) A white sphere appears and the user can locate it where they prefer, (ii) Pressing the Place button, the sphere turns green and keeps its position, indicating the first reference point, (iii) A second reference point is located in the same way, (iv) After the third point is located, the relative plane appears in the space.
• Figure 3: Trajectories superimposed to the real environment. (LEFT) The appearance of a trajectory tunnel before the beginning of the execution, with the functioning buttons and the slider. The tunnel appears completely red before the user starts moving. (RIGHT) The pictures show the appearance of the trajectory with the three confidence intervals (CIs).
• Figure 4: Feedback system showing the error-based feedback system. (LEFT) The spheres have different sizes and colours according to the error in their specific position along the trajectory. (RIGHT) At the end of the task, the system provides a snapshot of the movements executed, represented by a line that tracks the whole path of the hand.
• Figure 5: Steps to follow to execute a task with the HoloLens feedback: (i) Select the trajectory from the options; (ii) Position the trajectory on the Start Point using the blue button; (iii) Start the exercise with the Green button; (iv) Start moving trying to keep the hand in the red tunnel; (v) Execute the trajectory: the tunnel turns green if the execution is correct; (vi) Press the red button to stop.