Continual Learning: Forget-free Winning Subnetworks for Video Representations

TL;DR

FSO enhances higher-layer performance in TIL and FSCIL and lower-layer performance in VIL and lower-layer performance in VIL, and enhances higher-layer performance in TIL and FSCIL and lower-layer performance in VIL.

Abstract

Inspired by the Lottery Ticket Hypothesis (LTH), which highlights the existence of efficient subnetworks within larger, dense networks, a high-performing Winning Subnetwork (WSN) in terms of task performance under appropriate sparsity conditions is considered for various continual learning tasks. It leverages pre-existing weights from dense networks to achieve efficient learning in Task Incremental Learning (TIL) and Task-agnostic Incremental Learning (TaIL) scenarios. In Few-Shot Class Incremental Learning (FSCIL), a variation of WSN referred to as the Soft subnetwork (SoftNet) is designed to prevent overfitting when the data samples are scarce. Furthermore, the sparse reuse of WSN weights is considered for Video Incremental Learning (VIL). The use of Fourier Subneural Operator (FSO) within WSN is considered. It enables compact encoding of videos and identifies reusable subnetworks across varying bandwidths. We have integrated FSO into different architectural frameworks for continual learning, including VIL, TIL, and FSCIL. Our comprehensive experiments demonstrate FSO's effectiveness, significantly improving task performance at various convolutional representational levels. Specifically, FSO enhances higher-layer performance in TIL and FSCIL and lower-layer performance in VIL.

Figures (17)

• Figure 1: Concept Comparison: (a) Piggyback mallya2018piggyback, and SupSup wortsman2020supermasks find the optimal binary mask on a fixed backbone network a given task (b) PackNet mallya2018packnet and CLNP golkar2019continual forces the model to reuse all features and weights from previous subnetworks which cause bias in the transfer of knowledge (c) APD Yoon2020 selectively reuse and dynamically expand the dense network (d) Our WSN selectively reuse and dynamically expand subnetworks within a dense network. Green edges are reused weights.
• Figure 2: An illustration of Winning SubNetworks (WSN): (a) The top-c% weights$\hat{\bm{\theta}}_{s-1}$ at prior task are obtained, (b) In the forward pass of a new task, WSN selects the top-c% and reuses weights selected from prior tasks, (c) In the backward pass, WSN updates only non-used weights( ) while freezing reused weights( ), and (d) after several iterations of (b) and (c), we acquire again the top-c% weights$\hat{\bm{\theta}}_{s}$ including subsets of reused weights (green) for the new task.
• Figure 3: Forget-free Neural Implicit Representation with Fourier Subneural Operator (FSO) for Video Incremental Learning: Image-wise neural implicit representation taking frame and video (session $s$) indices as input and using a sparse Stems + NeRV Blocks with Fourier Subneural Operator (FSO) to output the whole image through multi-heads $H_N$ where $\tilde{\bm{v}}_s^t$ is a hidden representation. We denote frozen, reused, and trainable parameters in training at session 2. Note that each video representation is colored. In inference, we only need indices of session $s$ and frame $t$ and session mask (subnetwork).
• Figure 4: Residual Blocks (ResBlocks) with Fourier Subnerual Operator (FSO).
• Figure 5: The comparisons of WSN of 4 Conv (blue area) and 3 FC (gray area) with FSO (white area) in terms of Feature variances and high-frequency components: the (a) offers the variance of the feature map and the (b) provides $\Delta \log$ amplitudes at high-frequency ($1.0 \pi$).

Theorems & Definitions (4)

• Lemma 1
• Lemma 2
• Theorem 3
• Theorem 4