Towards Weakly Supervised End-to-end Learning for Long-video Action Recognition

TL;DR

This work tackles end-to-end long-video action recognition under only video-level supervision, where precise action intervals are unavailable and full clip-level labels are expensive to obtain. It introduces AdaptFocus, a three-module framework comprising Action Saliency Estimation (ASE), Action Focusing, and Clip Focusing, which jointly estimate per-action spike-actionness $a^k$ and salient times $\lambda^k$ to selectively train on cleaner clips via a self-paced weighting scheme. The approach is validated on three long-video datasets (Charades, Breakfast, MultiThumos) across CNN-based and Transformer-based backbones, showing that Weak (Ours) nearly matches Full (Clean) performance and significantly surpasses Weak (Noisy). Additionally, AdaptFocus enables a weakly supervised feature extraction pipeline (Weak-FEAT) that improves downstream long-video tasks such as Temporal Sentence Grounding, Complex Activity Recognition, and Temporal Action Segmentation, reducing the annotation burden while boosting robustness of long-video understanding. Overall, AdaptFocus offers a practical, scalable path toward robust, end-to-end weakly supervised long-video understanding and provides a useful feature extraction paradigm for a range of downstream tasks.

Abstract

Developing end-to-end action recognition models on long videos is fundamental and crucial for long-video action understanding. Due to the unaffordable cost of end-to-end training on the whole long videos, existing works generally train models on short clips trimmed from long videos. However, this ``trimming-then-training'' practice requires action interval annotations for clip-level supervision, i.e., knowing which actions are trimmed into the clips. Unfortunately, collecting such annotations is very expensive and prevents model training at scale. To this end, this work aims to build a weakly supervised end-to-end framework for training recognition models on long videos, with only video-level action category labels. Without knowing the precise temporal locations of actions in long videos, our proposed weakly supervised framework, namely AdaptFocus, estimates where and how likely the actions will occur to adaptively focus on informative action clips for end-to-end training. The effectiveness of the proposed AdaptFocus framework is demonstrated on three long-video datasets. Furthermore, for downstream long-video tasks, our AdaptFocus framework provides a weakly supervised feature extraction pipeline for extracting more robust long-video features, such that the state-of-the-art methods on downstream tasks are significantly advanced. We will release the code and models.

Figures (10)

• Figure 1: (a): End-to-end training action recognition models on long videos by sampling short clips, which requires full supervision of action categories and intervals; (b): Similar to (a), but the action interval annotations are unavailable. Such weak supervision will introduce noisy action labels that hurt model training; (c): Models trained under noisy weak supervision exhibit significant performance degradation compared to full supervision training. With our AdaptFocus framework, models can alleviate the influence of noises, and achieve comparable results to the models trained under clean full supervision.
• Figure 1: Action recognition results on Charades dataset. The mean average precision (mAP) metric is used. $F\times \tau$ denotes the number of frames$\times$stride between frames of clips. Full (Clean) means that the action interval annotations are used. Weak (Noisy) denotes that the all video-level actions are used as the labels of trimmed clips. Weak (Ours) denotes that the proposed AdaptFocus is applied during end-to-end training under weak supervision.
• Figure 2: Illustrations of the three modules in the proposed AdaptFocus framework.
• Figure 3: Temporal Sentence Grounding on Charades.
• Figure 4: Visualization of the temporal distribution of the ground-truth (black curve) and prediction (red curve) of actions in a video. Each sub-figure shows an action. (Zoom in for the best view)