Customized Mid-Air Gestures for Accessibility: A $B Recognizer for Multi-Dimensional Biosignal Gestures

TL;DR

The B-recognizer enables mid-air gesture recognition without needing expertise in biosignals or algorithms, and highlights the potential and feasibility of accessible biosignal interfaces.

Abstract

Biosignal interfaces, using sensors in, on, or around the body, promise to enhance wearables interaction and improve device accessibility for people with motor disabilities. However, biosignals are multi-modal, multi-dimensional, and noisy, requiring domain expertise to design input features for gesture classifiers. The \$B-recognizer enables mid-air gesture recognition without needing expertise in biosignals or algorithms. \$B resamples, normalizes, and performs dimensionality reduction to reduce noise and enhance signals relevant to the recognition. We tested \$B on a dataset of 26 participants with and 8 participants without upper-body motor disabilities performing personalized ability-based gestures. For two conditions (user-dependent, gesture articulation variability), \$B outperformed our comparison algorithms (traditional machine learning with expert features and deep learning), with > 95% recognition rate. For the user-independent condition, \$B and deep learning performed comparably for participants with disabilities. Our biosignal dataset is publicly available online. $B highlights the potential and feasibility of accessible biosignal interfaces.

Figures (6)

• Figure 1: Biosignals can have relevant (signals 1 and 2 in dark and light solid blue) or irrelevant (noise in dotted black) channels for recognition. Both relevant and irrelevant biosignals contain noise that must be reduced to improve recognition accuracy. Additionally, biosignals are often correlated. For example, signals 1 and 2 contain similar information in opposite orientations. As signal 1 increases in magnitude, signal 2 decreases in magnitude and vice versa. Decreasing signal correlations through dimensionality reduction reduces noise and improves recognition accuracy.
• Figure 2: Two different biosignals---(top, EMG) electromyography, measures muscle electrical activity; (bottom, IMU) inertial measurement unit, measures linear acceleration and rotation velocity---may have different sampling rates (left), and must be resampled (right) before comparisons are made. Note the two different measurement scales for the EMG signal (around $10^{-4}$ volts) and the IMU signal (around $2$ g).
• Figure 3: EMG (blue lines) and IMU (black lines) signals before (left) and after (right) normalization. The solid line represents relevant channels for recognition. The dotted line represents irrelevant noise channels. Before normalization (left), the EMG signal and noise are both significantly smaller in amplitude than the IMU signal and noise. After normalization (right), the EMG signal is now on the same scale as the IMU signal. Both EMG and IMU noise channels remain small in magnitude.
• Figure 4: In principal component analysis (PCA), high-dimensional signals (left) are decomposed into its principal components (middle). Multiplying them results in the signals being translated to a lower-dimensional latent space (right), where signal redundancy and noise have been reduced and signal variability is maximized. The original time-domain biosignals $D^T$ (left), or the transpose of matrix $D$, is a matrix with $n$ rows representing time and $c$ columns representing biosignal channels. The biosignal channels may represent multiple biosignals $b_1$ (blue), $b_2$ (grey). The principal components $U$ (middle), is a matrix with $c$ rows representing biosignal channels and $nPC$ columns representing the number of principal components to extract. $nPC$ is a parameter for the $B that must be tuned for a given dataset. In this figure, $nPC = 5$. Multiplying $D^T$ with $U$ will transform the high-dimensional signals in a lower-dimensional latent space (right) with $n$ rows representing time and $nPC$ columns representing the number of principal components. The right plot represents the transformed data plotted over the first two principal axes, which represent directions of greatest variance.
• Figure 5: Combined IMU and EMG sensors were placed on the participant's shoulders (trapezius, deltoid) and upper arms (biceps, triceps). The large black sensors on the forearms and wrists only sense IMU signals. The small yellow and blue sensors on the person's forearms (flexor, extensor) and hands (abductor pollicis brevis, first dorsal interosseous) only sense EMG signals. The placements were chosen to sense movement or muscle activation from the participant's upper body, including the fingers, arms, shoulders, neck, and head. This figure was adapted fromYamagami2023-xj.