Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Uncertainty-Based Evidential Fusion with Hybrid Multi-Head Attention for Weak-Supervised Temporal Action Localization

Yuanpeng He, Lijian Li, Tianxiang Zhan, Wenpin Jiao, Chi-Man Pun

TL;DR

This work tackles weakly supervised temporal action localization (WS-TAL) by addressing action-background ambiguity and feature redundancy. It introduces two key components: Hybrid Multi-Head Attention (HMHA) to refine RGB and optical-flow features and align their distributions, and Generalized Uncertainty-Based Evidential Fusion (GUEF) to fuse snippet-level evidences while explicitly modeling uncertainty to suppress background. The approach yields state-of-the-art WS-TAL performance on THUMOS14, with ablations confirming the critical roles of both HMHA and GUEF. Overall, the method improves foreground localization and classification under weak supervision, with code made publicly available to facilitate adoption and replication.

Abstract

Weakly supervised temporal action localization (WS-TAL) is a task of targeting at localizing complete action instances and categorizing them with video-level labels. Action-background ambiguity, primarily caused by background noise resulting from aggregation and intra-action variation, is a significant challenge for existing WS-TAL methods. In this paper, we introduce a hybrid multi-head attention (HMHA) module and generalized uncertainty-based evidential fusion (GUEF) module to address the problem. The proposed HMHA effectively enhances RGB and optical flow features by filtering redundant information and adjusting their feature distribution to better align with the WS-TAL task. Additionally, the proposed GUEF adaptively eliminates the interference of background noise by fusing snippet-level evidences to refine uncertainty measurement and select superior foreground feature information, which enables the model to concentrate on integral action instances to achieve better action localization and classification performance. Experimental results conducted on the THUMOS14 dataset demonstrate that our method outperforms state-of-the-art methods. Our code is available in \url{https://github.com/heyuanpengpku/GUEF/tree/main}.

Generalized Uncertainty-Based Evidential Fusion with Hybrid Multi-Head Attention for Weak-Supervised Temporal Action Localization

TL;DR

This work tackles weakly supervised temporal action localization (WS-TAL) by addressing action-background ambiguity and feature redundancy. It introduces two key components: Hybrid Multi-Head Attention (HMHA) to refine RGB and optical-flow features and align their distributions, and Generalized Uncertainty-Based Evidential Fusion (GUEF) to fuse snippet-level evidences while explicitly modeling uncertainty to suppress background. The approach yields state-of-the-art WS-TAL performance on THUMOS14, with ablations confirming the critical roles of both HMHA and GUEF. Overall, the method improves foreground localization and classification under weak supervision, with code made publicly available to facilitate adoption and replication.

Abstract

Weakly supervised temporal action localization (WS-TAL) is a task of targeting at localizing complete action instances and categorizing them with video-level labels. Action-background ambiguity, primarily caused by background noise resulting from aggregation and intra-action variation, is a significant challenge for existing WS-TAL methods. In this paper, we introduce a hybrid multi-head attention (HMHA) module and generalized uncertainty-based evidential fusion (GUEF) module to address the problem. The proposed HMHA effectively enhances RGB and optical flow features by filtering redundant information and adjusting their feature distribution to better align with the WS-TAL task. Additionally, the proposed GUEF adaptively eliminates the interference of background noise by fusing snippet-level evidences to refine uncertainty measurement and select superior foreground feature information, which enables the model to concentrate on integral action instances to achieve better action localization and classification performance. Experimental results conducted on the THUMOS14 dataset demonstrate that our method outperforms state-of-the-art methods. Our code is available in \url{https://github.com/heyuanpengpku/GUEF/tree/main}.
Paper Structure (11 sections, 16 equations, 1 figure, 2 tables)

This paper contains 11 sections, 16 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Hybrid Multi-Head Attention
  4. Deep Learning with Generalized Uncertainty-Based Evidential Fusion (GUEF)
  5. Learning and Inference
  6. Experiments and Results
  7. Datasets and Metrics
  8. Experiments Setup
  9. Performance Comparison with State-of-the-Art Models
  10. Ablation Study
  11. Conclusion

Figures (1)

  • Figure 1: The overall structure of our model.