Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Muscle to Text with MyoText: sEMG to Text via Finger Classification and Transformer-Based Decoding

Meghna Roy Chowdhury, Shreyas Sen, Yi Ding

TL;DR

MyoText tackles the challenge of keyboard-free text entry from sEMG by modeling typing as a motor–linguistic hierarchy. It first classifies active fingers from multichannel sEMG using a CNN–BiLSTM–Attention network, then applies ergonomic letter pooling to constrain candidate words, and finally uses a fine-tuned T5 transformer with constrained beam search to generate coherent sentences. On emg2qwerty with 30 participants, the system achieves 85.4% finger accuracy and 5.4% CER (6.5% WER), demonstrating robust cross-user generalization and the viability of on-device finger classification with edge-language decoding. The modular, physiologically grounded design reduces the decoding search space and leverages language models for long-range coherence, enabling practical, keyboard-free typing for AR/VR and ubiquitous computing. This work provides a scalable blueprint for wearable neural input that couples motor intent with linguistic reasoning under realistic hardware constraints.

Abstract

Surface electromyography (sEMG) provides a direct neural interface for decoding muscle activity and offers a promising foundation for keyboard-free text input in wearable and mixed-reality systems. Previous sEMG-to-text studies mainly focused on recognizing letters directly from sEMG signals, forming an important first step toward translating muscle activity into text. Building on this foundation, we present MyoText, a hierarchical framework that decodes sEMG signals to text through physiologically grounded intermediate stages. MyoText first classifies finger activations from multichannel sEMG using a CNN-BiLSTM-Attention model, applies ergonomic typing priors to infer letters, and reconstructs full sentences with a fine-tuned T5 transformer. This modular design mirrors the natural hierarchy of typing, linking muscle intent to language output and reducing the search space for decoding. Evaluated on 30 users from the emg2qwerty dataset, MyoText outperforms baselines by achieving 85.4% finger-classification accuracy, 5.4% character error rate (CER), and 6.5% word error rate (WER). Beyond accuracy gains, this methodology establishes a principled pathway from neuromuscular signals to text, providing a blueprint for virtual and augmented-reality typing interfaces that operate entirely without physical keyboards. By integrating ergonomic structure with transformer-based linguistic reasoning, MyoText advances the feasibility of seamless, wearable neural input for future ubiquitous computing environments.

From Muscle to Text with MyoText: sEMG to Text via Finger Classification and Transformer-Based Decoding

TL;DR

MyoText tackles the challenge of keyboard-free text entry from sEMG by modeling typing as a motor–linguistic hierarchy. It first classifies active fingers from multichannel sEMG using a CNN–BiLSTM–Attention network, then applies ergonomic letter pooling to constrain candidate words, and finally uses a fine-tuned T5 transformer with constrained beam search to generate coherent sentences. On emg2qwerty with 30 participants, the system achieves 85.4% finger accuracy and 5.4% CER (6.5% WER), demonstrating robust cross-user generalization and the viability of on-device finger classification with edge-language decoding. The modular, physiologically grounded design reduces the decoding search space and leverages language models for long-range coherence, enabling practical, keyboard-free typing for AR/VR and ubiquitous computing. This work provides a scalable blueprint for wearable neural input that couples motor intent with linguistic reasoning under realistic hardware constraints.

Abstract

Surface electromyography (sEMG) provides a direct neural interface for decoding muscle activity and offers a promising foundation for keyboard-free text input in wearable and mixed-reality systems. Previous sEMG-to-text studies mainly focused on recognizing letters directly from sEMG signals, forming an important first step toward translating muscle activity into text. Building on this foundation, we present MyoText, a hierarchical framework that decodes sEMG signals to text through physiologically grounded intermediate stages. MyoText first classifies finger activations from multichannel sEMG using a CNN-BiLSTM-Attention model, applies ergonomic typing priors to infer letters, and reconstructs full sentences with a fine-tuned T5 transformer. This modular design mirrors the natural hierarchy of typing, linking muscle intent to language output and reducing the search space for decoding. Evaluated on 30 users from the emg2qwerty dataset, MyoText outperforms baselines by achieving 85.4% finger-classification accuracy, 5.4% character error rate (CER), and 6.5% word error rate (WER). Beyond accuracy gains, this methodology establishes a principled pathway from neuromuscular signals to text, providing a blueprint for virtual and augmented-reality typing interfaces that operate entirely without physical keyboards. By integrating ergonomic structure with transformer-based linguistic reasoning, MyoText advances the feasibility of seamless, wearable neural input for future ubiquitous computing environments.
Paper Structure (35 sections, 2 equations, 15 figures, 11 tables)

This paper contains 35 sections, 2 equations, 15 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Electromyography (EMG)
  4. sEMG with Wearable Devices
  5. Applications of sEMG
  6. Challenges with sEMG Signals
  7. Prior Work in sEMG-to-Text Input
  8. Open Challenges in the State-of-the-Art
  9. Ergonomic Mapping of Letters to Fingers and the Variability
  10. Proposed Methodology: The MyoText Framework
  11. Data Preparation
  12. Finger-Level Classification
  13. Letter Pooling and Candidate Word Construction
  14. Sentence Prediction via Transformer-Based Decoding
  15. T5 Transformer.
  16. ...and 20 more sections

Figures (15)

  • Figure 1: MyoText. A physiologically-grounded modular sEMG-to-text decoding framework combining finger classification, ergonomic letter pooling, and sentence reconstruction.
  • Figure 2: EMG Sensing Modalities. Needle and surface EMG signal acquisition methods.
  • Figure 2: Letter to Finger Mapping
  • Figure 3: QWERTY Ergonomics and Finger Usage. (a) The standard QWERTY keyboard layout is depicted, showing the conventional finger-to-key assignments. Letters are distributed among eight active fingers; the thumbs (4, 5) are included for completeness. (b) Finger-based letter mapping. (c) Finger usage frequency, highlighting dominance of index and middle fingers in the emg2qwerty dataset.
  • Figure 3: Examples of Finger-to-Letter Mapping and Word Candidate Pooling.
  • ...and 10 more figures