FMI-TAL: Few-shot Multiple Instances Temporal Action Localization by Probability Distribution Learning and Interval Cluster Refinement

TL;DR

The paper tackles multi-instance, few-shot temporal action localization in long videos by introducing FMI-TAL, which combines a Spatial-Channel Relation Transformer with probability distribution learning and interval clustering. By modeling boundary likelihoods as probability distributions and refining predictions through selective cosine penalization and interval clustering, the method localizes multiple action instances without partitioning videos. Extensive experiments on ActivityNet1.3 and THUMOS14 demonstrate competitive performance and robust ablations validate the contribution of the SCR-Transformer, SCP, and the label-generation strategy. The approach offers a practical, end-to-end solution for real-world videos with sparse annotations and multiple actions.

Abstract

The present few-shot temporal action localization model can't handle the situation where videos contain multiple action instances. So the purpose of this paper is to achieve manifold action instances localization in a lengthy untrimmed query video using limited trimmed support videos. To address this challenging problem effectively, we proposed a novel solution involving a spatial-channel relation transformer with probability learning and cluster refinement. This method can accurately identify the start and end boundaries of actions in the query video, utilizing only a limited number of labeled videos. Our proposed method is adept at capturing both temporal and spatial contexts to effectively classify and precisely locate actions in videos, enabling a more comprehensive utilization of these crucial details. The selective cosine penalization algorithm is designed to suppress temporal boundaries that do not include action scene switches. The probability learning combined with the label generation algorithm alleviates the problem of action duration diversity and enhances the model's ability to handle fuzzy action boundaries. The interval cluster can help us get the final results with multiple instances situations in few-shot temporal action localization. Our model achieves competitive performance through meticulous experimentation utilizing the benchmark datasets ActivityNet1.3 and THUMOS14. Our code is readily available at https://github.com/ycwfs/FMI-TAL.

Figures (3)

• Figure 1: (a) Other methods need to split data first. (b) Our proposed method demonstrates the capability to localize multiple action instances within an untrimmed query video, utilizing a few trimmed support videos. This is achieved without necessitating dataset partitioning.
• Figure 2: Overview of our method. We first handle and integrate the extracted features by spatial-channel relation transformer. The enhanced features are fed into the Temporal Boundary Regression module to give probability distributions of action boundaries. All probability distributions of action boundaries are selected by the Temporal Segment Localization module to give the best results.
• Figure 3: Ablation of label generator