REST-HANDS: Rehabilitation with Egocentric Vision Using Smartglasses for Treatment of Hands after Surviving Stroke

TL;DR

The feasibility of using egocentric video from smart glasses for remote rehabilitation, specifically RayBan Stories, for remote hand rehabilitation is demonstrated, paving the way for further research.

Abstract

Stroke represents the third cause of death and disability worldwide, and is recognised as a significant global health problem. A major challenge for stroke survivors is persistent hand dysfunction, which severely affects the ability to perform daily activities and the overall quality of life. In order to regain their functional hand ability, stroke survivors need rehabilitation therapy. However, traditional rehabilitation requires continuous medical support, creating dependency on an overburdened healthcare system. In this paper, we explore the use of egocentric recordings from commercially available smart glasses, specifically RayBan Stories, for remote hand rehabilitation. Our approach includes offline experiments to evaluate the potential of smart glasses for automatic exercise recognition, exercise form evaluation and repetition counting. We present REST-HANDS, the first dataset of egocentric hand exercise videos. Using state-of-the-art methods, we establish benchmarks with high accuracy rates for exercise recognition (98.55%), form evaluation (86.98%), and repetition counting (mean absolute error of 1.33). Our study demonstrates the feasibility of using egocentric video from smart glasses for remote rehabilitation, paving the way for further research.

Figures (5)

• Figure 1: We propose REST-HANDS. Automatic remote hand rehabilitation assessment system for stroke survivors at home using egocentric vision through smartglasses. The images captured from the user's perspective are used to analyse the rehabilitation process with a spatio-temporal deep learning network to recognise exercises, evaluate exercise form and count repetitions. To assess the proposed idea, we create a first egocentric dataset with hand exercises for stroke patients and establish baseline results for three tasks by implementing and comparing different state-of-the-art architectures.
• Figure 2: Examples of frames from REST-HANDS dataset representing different exercises included in the dataset. Each exercise except Pushing Hands, which involves both hands, is performed and categorised separately for each hand resulting in 25 classes.
• Figure 3: Distribution of exercise classes and age of participants in REST-HANDS.
• Figure 4: Our methods for exercise recognition, form evaluation and repetition counting.
• Figure 5: Inference times for exercise recognition and pick detection with their accuracy. Each method is visualised as a circle whose size represents the number of parameters.