SMC-NCA: Semantic-guided Multi-level Contrast for Semi-supervised Temporal Action Segmentation

TL;DR

This work addresses semi-supervised temporal action segmentation in long untrimmed videos by introducing SMC-NCA, a framework that fuses temporal dynamics with action-specific semantic cues to learn robust frame-wise representations. A Semantic-guided Multi-level Contrast (SMC) learns intra- and inter-information variations via temporal and semantic branches, employing dynamic clustering-informed negative pairs and a positive pairing strategy that does not depend on noisy cluster labels. To curb over-segmentation in SSL, a Neighbourhood-Consistency-Aware (NCA) unit enforces spatially coherent neighbourhood distributions across frames, enabling strong segmentation with limited labelled data. The approach achieves consistent improvements over state-of-the-art methods on public human-action datasets (50Salads, GTEA, Breakfast) and demonstrates generalization to the Parkinson’s Disease Mouse Behaviour (PDMB) dataset, highlighting its applicability to long untrimmed videos beyond human actions.

Abstract

Semi-supervised temporal action segmentation (SS-TA) aims to perform frame-wise classification in long untrimmed videos, where only a fraction of videos in the training set have labels. Recent studies have shown the potential of contrastive learning in unsupervised representation learning using unlabelled data. However, learning the representation of each frame by unsupervised contrastive learning for action segmentation remains an open and challenging problem. In this paper, we propose a novel Semantic-guided Multi-level Contrast scheme with a Neighbourhood-Consistency-Aware unit (SMC-NCA) to extract strong frame-wise representations for SS-TAS. Specifically, for representation learning, SMC is first used to explore intra- and inter-information variations in a unified and contrastive way, based on action-specific semantic information and temporal information highlighting relations between actions. Then, the NCA module, which is responsible for enforcing spatial consistency between neighbourhoods centered at different frames to alleviate over-segmentation issues, works alongside SMC for semi-supervised learning (SSL). Our SMC outperforms the other state-of-the-art methods on three benchmarks, offering improvements of up to 17.8% and 12.6% in terms of Edit distance and accuracy, respectively. Additionally, the NCA unit results in significantly better segmentation performance in the presence of only 5% labelled videos. We also demonstrate the generalizability and effectiveness of the proposed method on our Parkinson Disease's Mouse Behaviour (PDMB) dataset. Code is available at https://github.com/FeixiangZhou/SMC-NCA.

Figures (10)

• Figure 1: Motivation of the proposed method. (a) ICC singhania2022iterative only samples representations with temporal information for unsupervised contrastive representation learning, where the construction of both positive (red boxes with the same label) and negative (boxes with different colours) pairs is guided by input feature clustering. It is susceptible to clustering errors (see Fig. \ref{['fig:evidence']}), leading to weaker frame-wise representations. (b) Our method leverages both temporal and semantic information to fully explore intra- and inter-information variations. We form positive pairs by considering the inherent temporal-semantic consistency of each frame, as well as three types of negative pairs based on the dynamic clustering of the original input, temporal and semantic features, thereby mitigating the adverse effects of clustering errors. (c) shows that our proposed method can learn more discriminative representations on the 50Salads dataset.
• Figure 2: Quantitative evidence that the quality of positive and negative pairs in ICC is affected by clustering errors. For both ICC and our SMC, we set the same batch size (5) and number of clusters (40), and the clustering accuracy is averaged over 50 epochs on the 50 Salads dataset by comparing the ground-truth labels with clustering labels. As observed, our SMC significantly improves the quality of positive pairs by leveraging the temporal-semantic consistency of each frame. The quality of negative pairs is also slightly enhanced by the dynamic clustering.
• Figure 3: Our proposed semi-supervised learning framework for temporal action segmentation. (a) Unsupervised representation learning. The pre-trained I3D features are fed into the long-term feature extractor $\textbf{T}$ and the semantic feature extractor $\textbf{S}$ to generate temporal and high-level semantic representations, respectively. Each kind of representation is sampled, followed by performing Semantic-guided Multi-level Contrast. (b) Semi-supervised learning. In stage 1, our Neighbourhood-Consistency-Aware unit $\textbf{G}$ works closely with SMC, and the classification layer $\textbf{C}$ can produce pseudo-labels (PL) used to guide the subsequent contrastive learning (stage 2). $\mathcal{L}_{ap}^{N}$, $\mathcal{L}_{aa}^{N}$ and $\mathcal{L}_{pp}^{N}$ denote the negative losses of three negative pairs which are driven by the clustering results. $\mathcal{L}_{ap}^{P}$ represents the positive loss without relying on the clustering labels. $\mathcal{L}_{nca}$ is generated in the NCA module. Note that T and C are two sub-networks of the C2F-TCN singhania2021coarsesinghania2022iterative backbone. The former is responsible for temporal feature extraction, while the latter is connected after the former to perform action classification. $\textbf{S}$ and $\textbf{G}$ are only used for training in order to facilitate learning and will not be used during the inference stage.
• Figure 4: Illustration of the proposed Neighbourhood-Consistency-Aware Unit. We randomly sample a frame $t$, and its neighbourhood is $N_{t}$ (left green dashed box). Then we can use frame $t^{*}$ with the same label to form the neighbourhood $N_{t^{*}}$. The two neighbourhoods are fed into the max-pooling layer to encode their feature distributions, followed by predicting the probability of having similar feature distributions by the MLP model. The same operation is applied to $N_{t}$ and $N_{t^{'}}$ centered at frames $t$ and $t^{'}$ with different labels.
• Figure 5: Performance of unsupervised representation learning using different loss functions during training on 50Salads.