Convex Combination Consistency between Neighbors for Weakly-supervised Action Localization

TL;DR

This work tackles weakly-supervised temporal action localization by addressing boundary localization challenges arising from subtle transitions between adjacent snippets. It introduces Convex Combination Consistency Between Neighbors (C$^3$BN), combining micro data augmentation via convex mixing of adjacent snippets with macro-micro consistency regularization across video semantics, snippet predictions, and snippet representations, including a bilateral contrastive mechanism aided by a projection head. The approach is general and can improve a range of WTAL baselines under video-level or point-level supervision, achieving state-of-the-art results on standard benchmarks and offering detailed ablations that confirm the contribution of each component. The method emphasizes exploiting fine-grained inter-snippet distinctions to produce more accurate action boundaries, with practical impact on WTAL performance and boundary robustness.

Abstract

Weakly-supervised temporal action localization (WTAL) intends to detect action instances with only weak supervision, e.g., video-level labels. The current~\textit{de facto} pipeline locates action instances by thresholding and grouping continuous high-score regions on temporal class activation sequences. In this route, the capacity of the model to recognize the relationships between adjacent snippets is of vital importance which determines the quality of the action boundaries. However, it is error-prone since the variations between adjacent snippets are typically subtle, and unfortunately this is overlooked in the literature. To tackle the issue, we propose a novel WTAL approach named Convex Combination Consistency between Neighbors (C$^3$BN). C$^3$BN consists of two key ingredients: a micro data augmentation strategy that increases the diversity in-between adjacent snippets by convex combination of adjacent snippets, and a macro-micro consistency regularization that enforces the model to be invariant to the transformations~\textit{w.r.t.} video semantics, snippet predictions, and snippet representations. Consequently, fine-grained patterns in-between adjacent snippets are enforced to be explored, thereby resulting in a more robust action boundary localization. Experimental results demonstrate the effectiveness of C$^3$BN on top of various baselines for WTAL with video-level and point-level supervisions. Code is at https://github.com/Qinying-Liu/C3BN.

Figures (7)

• Figure 1: Motivation of our method. Due to the vague distinctions between adjacent snippets, an under-performing model may produce similar activations for these snippets, resulting in incomplete/overcomplete proposals (see the upper T-CA). In this paper, we expect the model to correctly classify adjacent snippets, thereby localizing accurate boundaries (see the lower T-CAS).
• Figure 2: Overview of our method. We first perform convex combination on input snippet sequence $\boldsymbol F$ to produce the augmented one $\boldsymbol F'$. The procedure is denoted as micro data augmentation. Then we simultaneously feed the $\boldsymbol F$ and $\boldsymbol F'$ into the model and compute four regularization loss terms. We call it macro-micro consistency regularization.
• Figure 3: Illustration of micro data augmentation. The $\{ \boldsymbol f_t \}_{t=1}^{T-1}$ indicates the parent sequence and the $\{ \boldsymbol f'_t \}_{t=1}^{T-1}$ represents the child sequence.
• Figure 4: Illustration of snippet feature contrastive-consistency. It contains two parts: the child snippets are query while the parent ones are key (left); the parent snippets are query while the child ones are key (right).
• Figure 5: Qualitative results of T-CAS. "GT" denotes ground-truth annotation. "Base" denotes the T-CAS predicted by BaSNet while "+C$^3$BN" denotes that predicted by "BaSNet + C$^3$BN". The solid boxes indicate some noteworthy regions.