A Personalized Video-Based Hand Taxonomy: Application for Individuals with Spinal Cord Injury

TL;DR

This study aims to automatically identify the dominant distinct hand grasps in egocentric video using semantic clustering and offers researchers and clinicians an efficient tool for evaluating hand function, aiding sensitive assessments and tailored intervention plans.

Abstract

Hand function is critical for our interactions and quality of life. Spinal cord injuries (SCI) can impair hand function, reducing independence. A comprehensive evaluation of function in home and community settings requires a hand grasp taxonomy for individuals with impaired hand function. Developing such a taxonomy is challenging due to unrepresented grasp types in standard taxonomies, uneven data distribution across injury levels, and limited data. This study aims to automatically identify the dominant distinct hand grasps in egocentric video using semantic clustering. Egocentric video recordings collected in the homes of 19 individual with cervical SCI were used to cluster grasping actions with semantic significance. A deep learning model integrating posture and appearance data was employed to create a personalized hand taxonomy. Quantitative analysis reveals a cluster purity of 67.6% +- 24.2% with with 18.0% +- 21.8% redundancy. Qualitative assessment revealed meaningful clusters in video content. This methodology provides a flexible and effective strategy to analyze hand function in the wild. It offers researchers and clinicians an efficient tool for evaluating hand function, aiding sensitive assessments and tailored intervention plans.

Figures (9)

• Figure 1: The objective and pipeline of the paper: A) A person records a video with an egocentric camera in a naturalistic home environment. Deep learning methods were employed to extract B) hand poses and C) appearance information encoding hand-object association. D) Clustering algorithms were applied to categorize different grasp types using both hand poses and appearance information.
• Figure 2: MVC using postural and appearance information
• Figure 3: EEI and IEI for 2D pose, 3D pose, and appearance using GMM algorithm with tied covariance for A) HomeLab (n = 17 participants) and B) Home dataset (n = 19 participants), considering mean and standard errors.
• Figure 4: EEI and IEI using different weighting for postural information using GMM algorithm for Home dataset (n = 19 participants), considering mean and standard errors.
• Figure 5: Qualitative example of grasp clustering from participant 10 in the labelled subset of the Home dataset.