Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Solution for Temporal Action Localisation Task of Perception Test Challenge 2024

Yinan Han, Qingyuan Jiang, Hongming Mei, Yang Yang, Jinhui Tang

TL;DR

The paper tackles Temporal Action Localisation (TAL) in untrimmed videos within the Perception Test Challenge 2024. It presents a multimodal TAL pipeline that uses state-of-the-art video feature extractors (UMT and VideoMAEv2) and audio feature extractors (BEATs and CAV-MAE), augmented with overlapping labels from Something-SomethingV2 to improve generalization. The approach trains both multimodal and unimodal models and fuses their predictions with Weighted Box Fusion to enhance localisation robustness. The method achieves first place with a reported score of 0.5498, demonstrating the effectiveness of data augmentation and multimodal fusion for TAL in real-world perception tasks.

Abstract

This report presents our method for Temporal Action Localisation (TAL), which focuses on identifying and classifying actions within specific time intervals throughout a video sequence. We employ a data augmentation technique by expanding the training dataset using overlapping labels from the Something-SomethingV2 dataset, enhancing the model's ability to generalize across various action classes. For feature extraction, we utilize state-of-the-art models, including UMT, VideoMAEv2 for video features, and BEATs and CAV-MAE for audio features. Our approach involves training both multimodal (video and audio) and unimodal (video only) models, followed by combining their predictions using the Weighted Box Fusion (WBF) method. This fusion strategy ensures robust action localisation. our overall approach achieves a score of 0.5498, securing first place in the competition.

The Solution for Temporal Action Localisation Task of Perception Test Challenge 2024

TL;DR

The paper tackles Temporal Action Localisation (TAL) in untrimmed videos within the Perception Test Challenge 2024. It presents a multimodal TAL pipeline that uses state-of-the-art video feature extractors (UMT and VideoMAEv2) and audio feature extractors (BEATs and CAV-MAE), augmented with overlapping labels from Something-SomethingV2 to improve generalization. The approach trains both multimodal and unimodal models and fuses their predictions with Weighted Box Fusion to enhance localisation robustness. The method achieves first place with a reported score of 0.5498, demonstrating the effectiveness of data augmentation and multimodal fusion for TAL in real-world perception tasks.

Abstract

This report presents our method for Temporal Action Localisation (TAL), which focuses on identifying and classifying actions within specific time intervals throughout a video sequence. We employ a data augmentation technique by expanding the training dataset using overlapping labels from the Something-SomethingV2 dataset, enhancing the model's ability to generalize across various action classes. For feature extraction, we utilize state-of-the-art models, including UMT, VideoMAEv2 for video features, and BEATs and CAV-MAE for audio features. Our approach involves training both multimodal (video and audio) and unimodal (video only) models, followed by combining their predictions using the Weighted Box Fusion (WBF) method. This fusion strategy ensures robust action localisation. our overall approach achieves a score of 0.5498, securing first place in the competition.

Paper Structure

This paper contains 8 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Dataset
  4. Video Feature Extraction
  5. Audio Feature Extraction
  6. Prediction Fusion using WBF
  7. Experiments
  8. Conclusion

Figures (3)

  • Figure 1: the examples of Action Localisation annotations
  • Figure 2: Flowchart of our proposed method. The process begins with the Video Input and Audio Inpu, which are separately processed to extract features using pretrained models (UMT DBLP:conf/cvpr/LiuLWCSQ22, VideoMAEv2 DBLP:conf/cvpr/WangHZTHWWQ23 for video; BEATs DBLP:conf/icml/ChenW00T0CYW23, CAV-MAE DBLP:conf/iclr/GongRLHKKG23 for audio). The extracted Video Features and Audio Features are then fed into two models: the Multimodal Model combines both features for joint analysis, while the Unimodal Model uses only video features to evaluate the impact of audio information. Both models utilize ActionFormer DBLP:conf/eccv/ZhangWL22 for temporal action localisation. The outputs from these models are then WBF DBLP:journals/ivc/SolovyevWG21to produce the final action localisation results. This fusion leverages the strengths of both models, enhancing the accuracy and reliability of the action detection.
  • Figure 3: An illustration of the Weighted Box Fusion process in Temporal Action Localisation. Predictions from the multimodal and unimodal models are fused to produce a more accurate action localisation result.