Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Fitness Exercise Classification and Counting from Video Frames

Riccardo Riccio

TL;DR

A novel method for real-time exercise classification using a Bidirectional Long Short-Term Memory (BiLSTM) neural network that leverages invariant features, such as joint angles, alongside raw coordinates to improve generalization.

Abstract

This paper introduces a novel method for real-time exercise classification using a Bidirectional Long Short-Term Memory (BiLSTM) neural network. Existing exercise recognition approaches often rely on synthetic datasets, raw coordinate inputs sensitive to user and camera variations, and fail to fully exploit the temporal dependencies in exercise movements. These issues limit their generalizability and robustness in real-world conditions, where lighting, camera angles, and user body types vary. To address these challenges, we propose a BiLSTM-based model that leverages invariant features, such as joint angles, alongside raw coordinates. By using both angles and (x, y, z) coordinates, the model adapts to changes in perspective, user positioning, and body differences, improving generalization. Training on 30-frame sequences enables the BiLSTM to capture the temporal context of exercises and recognize patterns evolving over time. We compiled a dataset combining synthetic data from the InfiniteRep dataset and real-world videos from Kaggle and other sources. This dataset includes four common exercises: squat, push-up, shoulder press, and bicep curl. The model was trained and validated on these diverse datasets, achieving an accuracy of over 99% on the test set. To assess generalizability, the model was tested on 2 separate test sets representative of typical usage conditions. Comparisons with the previous approach from the literature are present in the result section showing that the proposed model is the best-performing one. The classifier is integrated into a web application providing real-time exercise classification and repetition counting without manual exercise selection. Demo and datasets are available at the following GitHub Repository: https://github.com/RiccardoRiccio/Fitness-AI-Trainer-With-Automatic-Exercise-Recognition-and-Counting.

Real-Time Fitness Exercise Classification and Counting from Video Frames

TL;DR

A novel method for real-time exercise classification using a Bidirectional Long Short-Term Memory (BiLSTM) neural network that leverages invariant features, such as joint angles, alongside raw coordinates to improve generalization.

Abstract

This paper introduces a novel method for real-time exercise classification using a Bidirectional Long Short-Term Memory (BiLSTM) neural network. Existing exercise recognition approaches often rely on synthetic datasets, raw coordinate inputs sensitive to user and camera variations, and fail to fully exploit the temporal dependencies in exercise movements. These issues limit their generalizability and robustness in real-world conditions, where lighting, camera angles, and user body types vary. To address these challenges, we propose a BiLSTM-based model that leverages invariant features, such as joint angles, alongside raw coordinates. By using both angles and (x, y, z) coordinates, the model adapts to changes in perspective, user positioning, and body differences, improving generalization. Training on 30-frame sequences enables the BiLSTM to capture the temporal context of exercises and recognize patterns evolving over time. We compiled a dataset combining synthetic data from the InfiniteRep dataset and real-world videos from Kaggle and other sources. This dataset includes four common exercises: squat, push-up, shoulder press, and bicep curl. The model was trained and validated on these diverse datasets, achieving an accuracy of over 99% on the test set. To assess generalizability, the model was tested on 2 separate test sets representative of typical usage conditions. Comparisons with the previous approach from the literature are present in the result section showing that the proposed model is the best-performing one. The classifier is integrated into a web application providing real-time exercise classification and repetition counting without manual exercise selection. Demo and datasets are available at the following GitHub Repository: https://github.com/RiccardoRiccio/Fitness-AI-Trainer-With-Automatic-Exercise-Recognition-and-Counting.

Paper Structure

This paper contains 32 sections, 9 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Human Pose Estimation for Exercise Tracking
  4. Automatic Exercise Classification
  5. Different Approaches to Automatic Exercise Classification
  6. 1. CNN + Soft Voting:
  7. 2. LSTM using x,y,z coordinates:
  8. 3. DNN on single frame + aggregation:
  9. Limitations of These Approaches and the Model Proposed in This Project
  10. Methodology
  11. Dataset
  12. Preprocessing Pipeline for Feature Creation
  13. Landmark and Features Extraction
  14. Models Training and Evaluations
  15. Loading and Preparing Data
  16. ...and 17 more sections

Figures (9)

  • Figure 1: topology of the 33 key points [2]
  • Figure 2: Images from the Datasets used for training and validation of the exercise classifier. (images are cropped due to screenshots, hence they appear to have different dimensions)
  • Figure 3: Images of the datasets used for additional testing (images are cropped due to screenshots)
  • Figure 4: model architecture
  • Figure 5: Confusion Matrices for LSTM and BiLSTM Models
  • ...and 4 more figures