Towards Activated Muscle Group Estimation in the Wild

TL;DR

This work tackles video-based Activated Muscle Group Estimation in the wild by introducing the MuscleMap dataset (15,004 clips, 135 activities, 20 muscle groups across four modalities) and a novel cross-modal architecture, TransM3E, that fuses RGB video and 2D skeleton data via Multi-Classification Tokens (MCT), cross-modal knowledge distillation (MCTKD), and fusion (MCTF). By explicitly evaluating generalization to unseen activities, the authors demonstrate that prior architectures struggle to generalize, and show that TransM3E achieves state-of-the-art performance on MuscleMap, outperforming baselines in both known and new activity types. The approach leverages a Transformer-based backbone (MViTv2) with K/D fusion strategies across modalities, enabling long-range reasoning essential for AMGE. The work advances practical video-based AMGE with broad applications in sports and rehabilitation, and provides a public dataset and codebase for further research.

Abstract

In this paper, we tackle the new task of video-based Activated Muscle Group Estimation (AMGE) aiming at identifying active muscle regions during physical activity in the wild. To this intent, we provide the MuscleMap dataset featuring >15K video clips with 135 different activities and 20 labeled muscle groups. This dataset opens the vistas to multiple video-based applications in sports and rehabilitation medicine under flexible environment constraints. The proposed MuscleMap dataset is constructed with YouTube videos, specifically targeting High-Intensity Interval Training (HIIT) physical exercise in the wild. To make the AMGE model applicable in real-life situations, it is crucial to ensure that the model can generalize well to numerous types of physical activities not present during training and involving new combinations of activated muscles. To achieve this, our benchmark also covers an evaluation setting where the model is exposed to activity types excluded from the training set. Our experiments reveal that the generalizability of existing architectures adapted for the AMGE task remains a challenge. Therefore, we also propose a new approach, TransM3E, which employs a multi-modality feature fusion mechanism between both the video transformer model and the skeleton-based graph convolution model with novel cross-modal knowledge distillation executed on multi-classification tokens. The proposed method surpasses all popular video classification models when dealing with both, previously seen and new types of physical activities. The database and code can be found at https://github.com/KPeng9510/MuscleMap.

Figures (6)

• Figure 1: Overview of the proposed MuscleMap dataset (Top) and the TransM$^3$E model (Bottom). Our dataset contains four data modalities, i.e., RGB, RGB difference (RGB Diff), optical flow, and 2D skeleton. PE and TF denote the patch embedding layer and the transformer block, respectively.
• Figure 2: An overview of the number of samples and the number of activity types per muscle region (@R), depicted at the top left and the top right.
• Figure 3: An overview of the proposed TransM$^3$E architecture.
• Figure 4: An overview of the modified GCN block with knowledge distillation MCT and classification MCT.
• Figure 5: Qualitative results for the MViTv2-S li2022mvitv2 and TransM$^3$E. GradCam selvaraju2017grad visualization is given.