Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment

TL;DR

This work tackles unsupervised temporal activity segmentation by introducing UFSA, a transformer-based framework that jointly leverages frame-level cues and segment-level cues through a frame-level encoder, a segment-level transcript decoder, and a frame-to-segment alignment module. It employs permutation-aware pseudo labels derived from temporal optimal transport to train all components without explicit action annotations, and it estimates transcripts from frame-level predictions to enable action permutations. Across four public datasets, UFSA achieves state-of-the-art or competitive results in MOF and F1-Score, demonstrating the effectiveness of incorporating high-level transcript information and permutation-aware alignment in unsupervised settings. The approach offers a scalable, annotation-free path to accurate activity segmentation with strong generalization across diverse tasks.

Abstract

This paper presents an unsupervised transformer-based framework for temporal activity segmentation which leverages not only frame-level cues but also segment-level cues. This is in contrast with previous methods which often rely on frame-level information only. Our approach begins with a frame-level prediction module which estimates framewise action classes via a transformer encoder. The frame-level prediction module is trained in an unsupervised manner via temporal optimal transport. To exploit segment-level information, we utilize a segment-level prediction module and a frame-to-segment alignment module. The former includes a transformer decoder for estimating video transcripts, while the latter matches frame-level features with segment-level features, yielding permutation-aware segmentation results. Moreover, inspired by temporal optimal transport, we introduce simple-yet-effective pseudo labels for unsupervised training of the above modules. Our experiments on four public datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop Assembly show that our approach achieves comparable or better performance than previous methods in unsupervised activity segmentation. Our code and dataset are available on our research website: https://retrocausal.ai/research/.

Figures (8)

• Figure 1: Prior works often use only frame-level cues via frame-level prediction modules (i.e., red) to predict framewise action classes. We adopt a segment-level prediction module and a frame-to-segment alignment module (i.e., green/blue), which exploit segment-level cues for permutation-aware results. Also, we introduce simple-yet-effective pseudo labels for unsupervised training.
• Figure 2: Our approach includes a frame-level prediction module (i.e., red) which extracts frame-level features $\boldsymbol{E}$ via a transformer encoder and uses temporal optimal transport to compute frame-level pseudo labels $\boldsymbol{Q}_f$ for unsupervised training. To exploit segment-level information, we utilize a segment-level prediction module (i.e., green), which extract segment-level features $\boldsymbol{D}$ via a transformer decoder, and a frame-to-segment alignment module (i.e., blue), which matches frame-level features $\boldsymbol{E}$ and segment-level features $\boldsymbol{D}$. In addition, we introduce segment-level pseudo labels $\boldsymbol{Q}_s$ and alignment-level pseudo labels $\boldsymbol{Q}_a$ for unsupervised training of the above modules.
• Figure 3: (a) Fixed-order prior distribution $\boldsymbol{M}_{\boldsymbol{A}}$. (b) Frame-level pseudo-label codes $\boldsymbol{Q}_f$.
• Figure 4: (a) Permutation-aware transcript $\boldsymbol{T}$. (b) Segment-level pseudo-label codes $\boldsymbol{Q}_s$.
• Figure 5: (a) Permutation-aware prior distribution $\boldsymbol{M}_{\boldsymbol{T}}$. (b) Alignment-level pseudo-label codes $\boldsymbol{Q}_a$.