Balancing Continual Learning and Fine-tuning for Human Activity Recognition

TL;DR

This work tackles continual learning for wearable-based HAR under limited labeled data by adapting two CSSL frameworks, CaSSLe and Kaizen, to enable continual representation learning and downstream classifier updates. A key contribution is the introduction of an adaptive importance coefficient $\lambda_{\mathrm{C}}$ to balance $\mathcal{L}^{\mathrm{KD}}_{\mathrm{C}}$ and new-task learning, yielding superior Final and Continual accuracy, particularly with BYOL as the SSL backbone. The results on the WISDM2019 dataset show Kaizen consistently outperforms CaSSLe and No Distill, illustrating the practicality of unified, continual learning for HAR with real-time adaptability. The findings highlight the value of progressive loss weighting in balancing retention and plasticity, enabling HAR systems to adapt to evolving user behaviors while mitigating catastrophic forgetting.

Abstract

Wearable-based Human Activity Recognition (HAR) is a key task in human-centric machine learning due to its fundamental understanding of human behaviours. Due to the dynamic nature of human behaviours, continual learning promises HAR systems that are tailored to users' needs. However, because of the difficulty in collecting labelled data with wearable sensors, existing approaches that focus on supervised continual learning have limited applicability, while unsupervised continual learning methods only handle representation learning while delaying classifier training to a later stage. This work explores the adoption and adaptation of CaSSLe, a continual self-supervised learning model, and Kaizen, a semi-supervised continual learning model that balances representation learning and down-stream classification, for the task of wearable-based HAR. These schemes re-purpose contrastive learning for knowledge retention and, Kaizen combines that with self-training in a unified scheme that can leverage unlabelled and labelled data for continual learning. In addition to comparing state-of-the-art self-supervised continual learning schemes, we further investigated the importance of different loss terms and explored the trade-off between knowledge retention and learning from new tasks. In particular, our extensive evaluation demonstrated that the use of a weighting factor that reflects the ratio between learned and new classes achieves the best overall trade-off in continual learning.

Figures (9)

• Figure 1: Performance comparison between different training methods. Models are trained using different self-supervised learning methods and knowledge distillation strategies on class-incremental WISDM2019. The top figure shows the average performance across the entire continual learning process, while the bottom figure shows the performance in the final evaluation.
• Figure 2: Average performance over tasks on WISDM2019. Comparison is drawn between training methods using different SSL algorithms across different tasks.
• Figure 3: Detailed breakdown of performance over tasks on WISDM2019. Fine-grained accuracy for every task is shown.
• Figure 4: Effects of constant importance coefficients. The plot compares the aggregate performance metrics of models trained with the importance coefficient for the classifier set to different constant values.
• Figure 5: Detailed breakdown of performance over tasks with constant importance coefficients. Fine-grained accuracy is shown for every additional task with different importance coefficients.