Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation

TL;DR

Gesture recognition for motor-impaired individuals faces distribution shifts and data scarcity. The paper presents Latent Embedding Exploitation (LEE), a replay-based few-shot continual learning method that uses a preserved gesture prior $z_c$ from control data and two adaptive embeddings to expand the latent space and learn unseen gestures from few examples, optimized with a learning objective $\mathcal{L} = \alpha\mathcal{L}_{ci} + \beta\mathcal{L}_{ii} + \mathcal{L}_{cls}$ and aided by memory replay. Experiments on SmartWatch Gesture and Motion Gesture datasets show that LEE often outperforms strong baselines and offers lower forgetting with fewer samples, while remaining robust to hyperparameters and source-domain variation. This approach enables personalized, natural gesture interfaces for motor-impaired users using wearable devices and opens paths toward online adaptation and broader inclusion in human-computer interaction.

Abstract

Hand gestures can provide a natural means of human-computer interaction and enable people who cannot speak to communicate efficiently. Existing hand gesture recognition methods heavily depend on pre-defined gestures, however, motor-impaired individuals require new gestures tailored to each individual's gesture motion and style. Gesture samples collected from different persons have distribution shifts due to their health conditions, the severity of the disability, motion patterns of the arms, etc. In this paper, we introduce the Latent Embedding Exploitation (LEE) mechanism in our replay-based Few-Shot Continual Learning (FSCL) framework that significantly improves the performance of fine-tuning a model for out-of-distribution data. Our method produces a diversified latent feature space by leveraging a preserved latent embedding known as gesture prior knowledge, along with intra-gesture divergence derived from two additional embeddings. Thus, the model can capture latent statistical structure in highly variable gestures with limited samples. We conduct an experimental evaluation using the SmartWatch Gesture and the Motion Gesture datasets. The proposed method results in an average test accuracy of 57.0%, 64.6%, and 69.3% by using one, three, and five samples for six different gestures. Our method helps motor-impaired persons leverage wearable devices, and their unique styles of movement can be learned and applied in human-computer interaction and social communication. Code is available at: https://github.com/riyadRafiq/wearable-latent-embedding-exploitation

Figures (6)

• Figure 1: (a) An individual lacking fine motor skills performs hand gestures. (b) Sensor-based gesture samples of two different participants including a control participant (top) and a motor-impaired participant (bottom). Data samples are more variable and noisy for a motor-impaired individual rather than a control participant. Blue, orange, and green lines are acceleration values along the x, y, and z-axis respectively.
• Figure 2: The complete framework containing the LEE mechanism. A latent embedding, $\mathbf{z_c}$ from the control population is preserved to work as gesture prior knowledge. In addition to it, two latent embeddings, $\mathbf{z_i}$ and $\mathbf{z_i^c}$ function to maintain intra-gesture divergence. The memory buffer saves the training samples from old gesture classes and provides them while training on a novel class.
• Figure 3: Test accuracies for a motor-impaired individual with Spinal cord injury (top row), an individual with Parkinson's disease (middle row), and a participant with Multiple sclerosis (bottom row) in a few-shot continual learning setting. The accuracy represents the total accuracy over all the gesture classes encountered trained with one, three, and five samples.
• Figure 4: Performance-forgetting scaled score for Gesture 1 (left) and Gesture 3 (right) for a motor-impaired individual with Spinal cord injury after six gestures are trained with five training examples.
• Figure 5: Accuracy for different hyperparameter values (left), number of participants (middle), and number of gestures (right) from the source domain when the preserved latent embedding is produced. We report the accuracy after six gestures are trained with five training examples.