SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay

TL;DR

SelfReplay tackles the domain shift problem that arises when self-supervised models pre-trained on unlabeled mobile-sensing data are fine-tuned across diverse users and devices. It introduces MetaSSL, a meta-learning-based pre-training stage that yields domain-adaptive weights, and ReplaySSL, an adaptation step that replays the meta-learned self-supervised task on the target domain using few-shot data before supervised fine-tuning. Across multiple mobile-sensing benchmarks and self-supervised objectives, SelfReplay achieves an average improvement of about 9.4 percentage points in F1-score over strong baselines and adapts on-device with modest overhead (e.g., under 10 seconds for SimCLR/Multi-Task on a smartphone). The approach is agnostic to the SSL objective and demonstrates robustness to temporal shifts, noisy domain labels, and varying model sizes, making it practical for end-user personalization on commodity devices.

Abstract

Self-supervised learning has emerged as a method for utilizing massive unlabeled data for pre-training models, providing an effective feature extractor for various mobile sensing applications. However, when deployed to end-users, these models encounter significant domain shifts attributed to user diversity. We investigate the performance degradation that occurs when self-supervised models are fine-tuned in heterogeneous domains. To address the issue, we propose SelfReplay, a few-shot domain adaptation framework for personalizing self-supervised models. SelfReplay proposes self-supervised meta-learning for initial model pre-training, followed by a user-side model adaptation by replaying the self-supervision with user-specific data. This allows models to adjust their pre-trained representations to the user with only a few samples. Evaluation with four benchmarks demonstrates that SelfReplay outperforms existing baselines by an average F1-score of 8.8%p. Our on-device computational overhead analysis on a commodity off-the-shelf (COTS) smartphone shows that SelfReplay completes adaptation within an unobtrusive latency (in three minutes) with only a 9.54% memory consumption, demonstrating the computational efficiency of the proposed method.

Figures (9)

• Figure 1: Illustration of domain shift on a self-supervised model pre-trained in one domain and fine-tuned in another for human activity recognition. Fine-tuning on the target domain results in a 19.6% F1-score drop (87% vs. 67.4%).
• Figure 2: A comparison between the standard pre-training and fine-tuning (top) and SelfReplay (bottom). Components in the grey box are performed in the source domain, while those in the blue box are performed in the target domain.
• Figure 3: Overview of SelfReplay. The model is first pre-trained through MetaSSL in the source domain to develop domain adaptability, followed by ReplaySSL to adapt the model to the target domain.
• Figure 4: Average F1-scores of SelfReplay and the baselines across different shot numbers (1, 2, 5, and 10).
• Figure 5: Average F1-scores of SelfReplay based on different self-supervised learning methods (SimCLR, CPC, and multi-task learning) across different shot numbers (1, 2, 5, and 10).