Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CPEP: Contrastive Pose-EMG Pre-training Enhances Gesture Generalization on EMG Signals

Wenhui Cui, Christopher Sandino, Hadi Pouransari, Ran Liu, Juri Minxha, Ellen Zippi, Aman Verma, Anna Sedlackova, Erdrin Azemi, Behrooz Mahasseni

TL;DR

This work tackles gesture recognition from noisy EMG signals and the problem of generalizing to unseen gestures. It introduces Contrastive Pose-EMG Pre-training (CPEP), a two-stage approach that first learns robust unimodal representations via MAE and then aligns EMG to high-quality pose embeddings using a contrastive objective, with the pose encoder frozen during alignment. By leveraging pose as an anchor, CPEP produces pose-informed EMG embeddings that enable zero-shot classification and improved in-distribution performance, outperforming several baselines by substantial margins on unseen gestures. The method demonstrates strong generalization on a large public EMG–pose dataset and establishes a foundation for multi-modal biosignal applications with minimal task-specific tuning.

Abstract

Hand gesture classification using high-quality structured data such as videos, images, and hand skeletons is a well-explored problem in computer vision. Leveraging low-power, cost-effective biosignals, e.g. surface electromyography (sEMG), allows for continuous gesture prediction on wearables. In this paper, we demonstrate that learning representations from weak-modality data that are aligned with those from structured, high-quality data can improve representation quality and enables zero-shot classification. Specifically, we propose a Contrastive Pose-EMG Pre-training (CPEP) framework to align EMG and pose representations, where we learn an EMG encoder that produces high-quality and pose-informative representations. We assess the gesture classification performance of our model through linear probing and zero-shot setups. Our model outperforms emg2pose benchmark models by up to 21% on in-distribution gesture classification and 72% on unseen (out-of-distribution) gesture classification.

CPEP: Contrastive Pose-EMG Pre-training Enhances Gesture Generalization on EMG Signals

TL;DR

This work tackles gesture recognition from noisy EMG signals and the problem of generalizing to unseen gestures. It introduces Contrastive Pose-EMG Pre-training (CPEP), a two-stage approach that first learns robust unimodal representations via MAE and then aligns EMG to high-quality pose embeddings using a contrastive objective, with the pose encoder frozen during alignment. By leveraging pose as an anchor, CPEP produces pose-informed EMG embeddings that enable zero-shot classification and improved in-distribution performance, outperforming several baselines by substantial margins on unseen gestures. The method demonstrates strong generalization on a large public EMG–pose dataset and establishes a foundation for multi-modal biosignal applications with minimal task-specific tuning.

Abstract

Hand gesture classification using high-quality structured data such as videos, images, and hand skeletons is a well-explored problem in computer vision. Leveraging low-power, cost-effective biosignals, e.g. surface electromyography (sEMG), allows for continuous gesture prediction on wearables. In this paper, we demonstrate that learning representations from weak-modality data that are aligned with those from structured, high-quality data can improve representation quality and enables zero-shot classification. Specifically, we propose a Contrastive Pose-EMG Pre-training (CPEP) framework to align EMG and pose representations, where we learn an EMG encoder that produces high-quality and pose-informative representations. We assess the gesture classification performance of our model through linear probing and zero-shot setups. Our model outperforms emg2pose benchmark models by up to 21% on in-distribution gesture classification and 72% on unseen (out-of-distribution) gesture classification.

Paper Structure

This paper contains 15 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Problem Definition
  4. Unimodal Encoder Pre‐training
  5. Contrastive Pose-EMG Pre-training (CPEP)
  6. Evaluation Protocols
  7. Experiments and Results
  8. Data
  9. Gesture Classification Results
  10. Ablation Study
  11. Discussion and Conclusion
  12. Details of Gestures
  13. In-distribution (4 classes).
  14. Unseen (4 classes).
  15. Implementation Details

Figures (4)

  • Figure 1: Overview of Contrastive Pose–EMG Pre-training (CPEP) framework. (a). MAE pre-trains EMG and pose encoders; (b). CPEP freezes the pose encoder and aligns EMG to pose via contrastive learning, enabling (c). zero-shot gesture classification by retrieval over frozen pose embeddings.
  • Figure 2: Classification accuracy on in-distribution and unseen gestures. Linear probing is reported for all methods; zero-shot is reported only for CPEP (other methods do not support zero-shot).
  • Figure 3: Ablation of MAE pre-training setup: temporal patch length $S$ and mask ratio $r$.
  • Figure 4: Example visualizations of gestures used in gesture classification tasks.

Theorems & Definitions (2)

  • Definition 1
  • Definition 2