Hierarchical Vector Quantization for Unsupervised Action Segmentation

TL;DR

This work tackles unsupervised temporal action segmentation in long videos by introducing Hierarchical Vector Quantization (HVQ), a two-level vector-quantization framework that learns fine-to-coarse action clusters via codebooks $Z$ and $Q$. The model is trained end-to-end with reconstruction and commitment losses, updates prototypes online, and employs FIFA decoding for temporally smoothed inference, without relying on pseudo-labels. It also introduces a Jensen-Shannon Distance-based metric to quantify segment-length bias, and demonstrates state-of-the-art results across Breakfast, YouTube Instructional, and IKEA ASM, with improved recall and F1 and reduced length-bias. The approach shows that hierarchical clustering better captures intra-action variation and yields stable cross-video action representations, offering practical benefits for fully unsupervised temporal understanding in diverse domains.

Abstract

In this work, we address unsupervised temporal action segmentation, which segments a set of long, untrimmed videos into semantically meaningful segments that are consistent across videos. While recent approaches combine representation learning and clustering in a single step for this task, they do not cope with large variations within temporal segments of the same class. To address this limitation, we propose a novel method, termed Hierarchical Vector Quantization (HVQ), that consists of two subsequent vector quantization modules. This results in a hierarchical clustering where the additional subclusters cover the variations within a cluster. We demonstrate that our approach captures the distribution of segment lengths much better than the state of the art. To this end, we introduce a new metric based on the Jensen-Shannon Distance (JSD) for unsupervised temporal action segmentation. We evaluate our approach on three public datasets, namely Breakfast, YouTube Instructional and IKEA ASM. Our approach outperforms the state of the art in terms of F1 score, recall and JSD.

Figures (10)

• Figure 1: Given a set of long untrimmed video sequences (first row), the goal is to segment each video and cluster the segments across all videos. Our approach for hierarchical vector quantization learns two codebooks ${\bf Z}$ (second row) and ${\bf Q}$ (third row), which results in a hierarchical representation of actions.
• Figure 2: Overview of our model. The encoder takes as input a video $X_m$ and encodes every frame into an embedding vector $e_{mt}$. Every $e_{mt}$ is associated with the closest prototype $z_j \in {\bf Z}$. The prototypes $z_j$ are then associated with the closest prototype $q_i \in {\bf Q}$. The association of a frame to $\bf Q$ via ${\bf Z}$ defines the cluster assignment.
• Figure 3: Histograms of the segment lengths across the whole milk activity for ground-truth, HVQ, TOT and ASOT. Segments longer than $580$ frames are in the last bin.
• Figure 4: Segmentation results for a sample from the Breakfast dataset ( P22_friedegg). Our approach delivers highly consistent results across multiple videos (V1, V2, V3, V4) recorded from different cameras, but with the same ground truth.
• Figure 5: Segmentation results for video P23_webcam01_pancake of Breakfast.