Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Action Recognition Using Temporal Shift Module and Ensemble Learning

Anh-Kiet Duong, Petra Gomez-Krämer

TL;DR

The paper addresses multi-modal action recognition with a small dataset (2,000 training videos, 500 test videos across 20 actions) and demonstrates that a thermal IR–focused approach using Temporal Shift Module (TSM) can achieve state-of-the-art performance. It combines transfer learning and fine-tuning with backbone ensembles (ResNeSt-269 for RGB; ResNeSt-269 and ResNeXt-101-64×4d for thermal IR) and fuses predictions via a weighted softmax ensemble, yielding a perfect Top-1/Top-5 score of 1.000 on the test set. The results highlight the robustness and efficiency of single-modality thermal IR models when paired with TSM and ensemble fusion, while also discussing trade-offs with computational cost for high-performing backbones. Overall, the work provides a strong benchmark for Track 3 of the ICPR 2024 challenge and resources for extending multi-modal action recognition with targeted modality focus and ensemble strategies.

Abstract

This paper presents the first-rank solution for the Multi-Modal Action Recognition Challenge, part of the Multi-Modal Visual Pattern Recognition Workshop at the \acl{ICPR} 2024. The competition aimed to recognize human actions using a diverse dataset of 20 action classes, collected from multi-modal sources. The proposed approach is built upon the \acl{TSM}, a technique aimed at efficiently capturing temporal dynamics in video data, incorporating multiple data input types. Our strategy included transfer learning to leverage pre-trained models, followed by meticulous fine-tuning on the challenge's specific dataset to optimize performance for the 20 action classes. We carefully selected a backbone network to balance computational efficiency and recognition accuracy and further refined the model using an ensemble technique that integrates outputs from different modalities. This ensemble approach proved crucial in boosting the overall performance. Our solution achieved a perfect top-1 accuracy on the test set, demonstrating the effectiveness of the proposed approach in recognizing human actions across 20 classes. Our code is available online https://github.com/ffyyytt/TSM-MMVPR.

Action Recognition Using Temporal Shift Module and Ensemble Learning

TL;DR

The paper addresses multi-modal action recognition with a small dataset (2,000 training videos, 500 test videos across 20 actions) and demonstrates that a thermal IR–focused approach using Temporal Shift Module (TSM) can achieve state-of-the-art performance. It combines transfer learning and fine-tuning with backbone ensembles (ResNeSt-269 for RGB; ResNeSt-269 and ResNeXt-101-64×4d for thermal IR) and fuses predictions via a weighted softmax ensemble, yielding a perfect Top-1/Top-5 score of 1.000 on the test set. The results highlight the robustness and efficiency of single-modality thermal IR models when paired with TSM and ensemble fusion, while also discussing trade-offs with computational cost for high-performing backbones. Overall, the work provides a strong benchmark for Track 3 of the ICPR 2024 challenge and resources for extending multi-modal action recognition with targeted modality focus and ensemble strategies.

Abstract

This paper presents the first-rank solution for the Multi-Modal Action Recognition Challenge, part of the Multi-Modal Visual Pattern Recognition Workshop at the \acl{ICPR} 2024. The competition aimed to recognize human actions using a diverse dataset of 20 action classes, collected from multi-modal sources. The proposed approach is built upon the \acl{TSM}, a technique aimed at efficiently capturing temporal dynamics in video data, incorporating multiple data input types. Our strategy included transfer learning to leverage pre-trained models, followed by meticulous fine-tuning on the challenge's specific dataset to optimize performance for the 20 action classes. We carefully selected a backbone network to balance computational efficiency and recognition accuracy and further refined the model using an ensemble technique that integrates outputs from different modalities. This ensemble approach proved crucial in boosting the overall performance. Our solution achieved a perfect top-1 accuracy on the test set, demonstrating the effectiveness of the proposed approach in recognizing human actions across 20 classes. Our code is available online https://github.com/ffyyytt/TSM-MMVPR.

Paper Structure

This paper contains 17 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Video action recognition
  4. Deep learning architectures
  5. Ensemble learning
  6. Action recognition method
  7. TSM
  8. Backbone models
  9. Data modality
  10. Training Strategy
  11. Configuration
  12. Ensemble learning
  13. Results
  14. Task description
  15. Validation
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of the prediction process in the proposed solution.
  • Figure 2: Comparison of RGB and thermal IR images from the same frame of the "ride a bike" class.
  • Figure 3: Validation accuracy of the ResNeSt269 and ResNeXt101 64x4d backbone models with thermal IR and RGB input on the validation set.