Examining Changes in Internal Representations of Continual Learning Models Through Tensor Decomposition

TL;DR

The paper tackles representational forgetting in continual learning by proposing a representation-based evaluation framework that tracks internal model activations across tasks and time. It applies Tensor Component Analysis (TCA) to three-way tensors of activations, inputs, and snapshots to uncover learning dynamics, neuron specialization, and filter evolution across architectures and CL strategies. Key findings show that while the approach can reflect relative performance differences, it struggles to reveal clear clusters of specialized neurons or interpretable filter dynamics, indicating both limitations and potential for refinement. The work contributes a novel interpretability framework for CL dynamics and highlights practical considerations and future directions for scalable, mechanistic analyses of internal representations in continual learning.

Abstract

Continual learning (CL) has spurred the development of several methods aimed at consolidating previous knowledge across sequential learning. Yet, the evaluations of these methods have primarily focused on the final output, such as changes in the accuracy of predicted classes, overlooking the issue of representational forgetting within the model. In this paper, we propose a novel representation-based evaluation framework for CL models. This approach involves gathering internal representations from throughout the continual learning process and formulating three-dimensional tensors. The tensors are formed by stacking representations, such as layer activations, generated from several inputs and model `snapshots', throughout the learning process. By conducting tensor component analysis (TCA), we aim to uncover meaningful patterns about how the internal representations evolve, expecting to highlight the merits or shortcomings of examined CL strategies. We conduct our analyses across different model architectures and importance-based continual learning strategies, with a curated task selection. While the results of our approach mirror the difference in performance of various CL strategies, we found that our methodology did not directly highlight specialized clusters of neurons, nor provide an immediate understanding the evolution of filters. We believe a scaled down version of our approach will provide insight into the benefits and pitfalls of using TCA to study continual learning dynamics.

Figures (13)

• Figure 1: Overview of the proposed methodology. (A) We train a model with a CL method with curated tasks. (B) We assume access to 'snapshots' of the model taken throughout training on all tasks. At inference time, for a given dataset class, we feed its corresponding representative inputs into each of the snapshots and, for a layer of choice, gather the resulting activation tensors. We flatten the activations into a vector and stack them for all snapshots to obtain a matrix. We repeat this process for all inputs and stack the matrices to obtain a tensor. (C) We conduct tensor component analysis (TCA) with rank $n$ to obtain $n$ components. (D) We plot $n$ components, each of which consists of three factors: factors selecting for certain activations (green), factors selecting for certain inputs (blue), and factors describing a temporal activity (red). (E) We conduct model masking experiments to verify our observations.
• Figure 2: Layer-wise reconstruction errors with shaded standard deviations for ResNet50 (left), CvT13 (middle), and DeiTSmall (right) models. Note: The y-axis scaling varies between plots for clearer visualization.
• Figure 3: Example of tensor component analysis on an activations tensor: two rank-13 TCA plots of activation tensors from DeitSmall trained on Split-CIFAR-10, permuted to order factors that are best aligned. The similarity score between the two is 0.58. Left-TCA plot of activations from the MAS strategy, with a reconstruction error of 0.82. Right-TCA plot of activation from the MAS Replay strategy, with a reconstruction error of 0.85. Despite the similar performance of the TCA models, as demonstrated in Figure \ref{['fig:layer_errors']}, the TCA models do not have a high similarity.
• Figure 4: Example of tensor component analysis on an optimized images for filters tensor: two rank-20 TCA plots of optimized images for filters tensors from ResNet50 trained on Split-CIFAR-10. Left-TCA plot of filter images from the Naive strategy, with a reconstruction error of 0.24. Right-TCA plot of activation from the WSN strategy, with a reconstruction error of 0.19.
• Figure A.5: Visualization of tSimCNE embeddings, the computed centroids for each class (marked by crosses), and the convex hull constructed using QuickHull algorithm for CIFAR-10 (left), CIFAR-100 superclasses (middle), and CIFAR-100 (right) datasets.