Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Video Class-Incremental Learning via Deep Embedded Clustering Management

Nattapong Kurpukdee, Adrian G. Bors

TL;DR

The paper addresses unsupervised video class-incremental learning (uVCIL) by proposing a memory-driven framework that continually builds deep embedded clusters from unlabeled video features and reuses past clusters to transfer knowledge without labels. It introduces two variants, uVCIL-CLU and uVCIL-CLU-RBF, that rely on fixed feature extractors, deep clustering to generate pseudo-labels, and an RBF-based or linear head with focal loss to handle cluster imbalance, all supported by per-cluster memory buffers for replay. Experiments on UCF101, HMDB51, and SSv2 demonstrate that the proposed approach outperforms adapted supervised baselines, achieving strong average cluster accuracy while maintaining computational efficiency. The work advances practical unlabeled continual video learning with a scalable memory-enabled clustering strategy and a reusable evaluation protocol.

Abstract

Unsupervised video class incremental learning (uVCIL) represents an important learning paradigm for learning video information without forgetting, and without considering any data labels. Prior approaches have focused on supervised class-incremental learning, relying on using the knowledge of labels and task boundaries, which is costly, requires human annotation, or is simply not a realistic option. In this paper, we propose a simple yet effective approach to address the uVCIL. We first consider a deep feature extractor network, providing a set of representative video features during each task without assuming any class or task information. We then progressively build a series of deep clusters from the extracted features. During the successive task learning, the model updated from the previous task is used as an initial state in order to transfer knowledge to the current learning task. We perform in-depth evaluations on three standard video action recognition datasets, including UCF101, HMDB51, and Something-to-Something V2, by ignoring the labels from the supervised setting. Our approach significantly outperforms other baselines on all datasets.

Unsupervised Video Class-Incremental Learning via Deep Embedded Clustering Management

TL;DR

The paper addresses unsupervised video class-incremental learning (uVCIL) by proposing a memory-driven framework that continually builds deep embedded clusters from unlabeled video features and reuses past clusters to transfer knowledge without labels. It introduces two variants, uVCIL-CLU and uVCIL-CLU-RBF, that rely on fixed feature extractors, deep clustering to generate pseudo-labels, and an RBF-based or linear head with focal loss to handle cluster imbalance, all supported by per-cluster memory buffers for replay. Experiments on UCF101, HMDB51, and SSv2 demonstrate that the proposed approach outperforms adapted supervised baselines, achieving strong average cluster accuracy while maintaining computational efficiency. The work advances practical unlabeled continual video learning with a scalable memory-enabled clustering strategy and a reusable evaluation protocol.

Abstract

Unsupervised video class incremental learning (uVCIL) represents an important learning paradigm for learning video information without forgetting, and without considering any data labels. Prior approaches have focused on supervised class-incremental learning, relying on using the knowledge of labels and task boundaries, which is costly, requires human annotation, or is simply not a realistic option. In this paper, we propose a simple yet effective approach to address the uVCIL. We first consider a deep feature extractor network, providing a set of representative video features during each task without assuming any class or task information. We then progressively build a series of deep clusters from the extracted features. During the successive task learning, the model updated from the previous task is used as an initial state in order to transfer knowledge to the current learning task. We perform in-depth evaluations on three standard video action recognition datasets, including UCF101, HMDB51, and Something-to-Something V2, by ignoring the labels from the supervised setting. Our approach significantly outperforms other baselines on all datasets.
Paper Structure (10 sections, 3 equations, 3 figures, 2 tables)

This paper contains 10 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. uVCIL methodology
  3. Problem Setup
  4. Assigning Pseudo-labels through deep clustering
  5. Linear cluster self-supervised learning
  6. Experimental Results
  7. Implementation Details
  8. uVCIL results on UCF101, HMDB51 and SSv2
  9. Ablation Study
  10. Conclusion

Figures (3)

  • Figure 1: Overview of uVCIL. A video dataset will be divided into a series of tasks $\{ \tau_1,\tau_2,\ldots,\tau_k \}$. Each task data will feeds into a deep feature extractor $g(\cdot)$, which is used to extract the video features. After that, each task $\tau_k$, a fixed number of deep embedded clusters is formed, and each cluster has a memory buffer associated in order to store $N$ feature vector for that cluster. For every new task, the feature from the memory buffers is reused through memory replay to tackle forgetting. Moreover, novel data will form new clusters, contributing to the augmentation of the stored information.
  • Figure 2: uVCIL results on UCF101, HMDB51 and SSv2 for the first fold data based on ResNet-34 feature extractor.
  • Figure 3: The visualization of cluster center along with their memory buffer after all subsequence learning task. This figure is best viewed in colour, $+$ represents the cluster centre, and a number represents the cluster ID. We visualize by using t-SNE for feature reduction to 2-Dimensions with a perplexity of 40.