Scaling laws in wearable human activity recognition

TL;DR

The paper investigates whether principled scaling laws connect pre-training data, model capacity, and performance in wearable HAR under data- and compute-constrained conditions. Using a Masked Autoencoder with a Vision Transformer encoder pre-trained on the Extrasensory dataset and evaluated via linear probing on UCI HAR and WISDM benchmarks, it shows that the pre-training loss $L$ scales as a power-law with data hours $D$ and parameter count $P$, i.e., $L \propto D^{-\alpha}$ and $L \propto P^{-\beta}$, and that the exponent is markedly larger when new users are added (data diversity) than when data is increased per user; larger models are needed to exploit more pre-training data. It also demonstrates that downstream performance improves in lockstep with pre-training data scale and model capacity, motivating larger, more diverse pre-training and careful allocation of compute and data collection. The work suggests revisiting prior under-parameterized HAR models and discusses practical paths to deployment, such as teacher-student distillation for on-device use.

Abstract

Many deep architectures and self-supervised pre-training techniques have been proposed for human activity recognition (HAR) from wearable multimodal sensors. Scaling laws have the potential to help move towards more principled design by linking model capacity with pre-training data volume. Yet, scaling laws have not been established for HAR to the same extent as in language and vision. By conducting an exhaustive grid search on both amount of pre-training data and Transformer architectures, we establish the first known scaling laws for HAR. We show that pre-training loss scales with a power law relationship to amount of data and parameter count and that increasing the number of users in a dataset results in a steeper improvement in performance than increasing data per user, indicating that diversity of pre-training data is important, which contrasts to some previously reported findings in self-supervised HAR. We show that these scaling laws translate to downstream performance improvements on three HAR benchmark datasets of postures, modes of locomotion and activities of daily living: UCI HAR and WISDM Phone and WISDM Watch. Finally, we suggest some previously published works should be revisited in light of these scaling laws with more adequate model capacities.

Figures (7)

• Figure 1: Masked Autoencoder adapted for accelerometer and gyroscope. During pre-training, a random subset of accelerometer and gyroscope patches are masked out. Non-masked patches are passed to the encoder and the mask tokens are re-introduced after the encoder. The encoded patches and mask tokens are then processed by a small decoder trained to reconstruct the original input sequence.
• Figure 2: Pre-training test loss vs data size (hours). We fit a power law to each fold and sampling strategy. Equations for each power law can be found in \ref{['pretrain-data-law-table']}.
• Figure 3: Pre-training test loss vs model capacity (number of parameters) and associated power law fit and equation.
• Figure 4: Best linear F1 scores vs pre-training dataset size (hours). Each point represents the best F1 score corresponding to a pre-training fold and data size. The best score is chosen from 27 runs consisting of the 9 encoder architectures and 3 learning rates in our search space.
• Figure 5: Best linear F1 scores vs model capacity (number of parameters). Each point represents the best F1 score corresponding to an encoder architecture (width and depth). The best score is chosen from all data sizes and learning rates. We indicate the width (mlp hidden dim) by color. At 5M or 20M parameters we have two models that are the same size, with one wider and shallower (5 blocks) and the other narrower and deeper (20 blocks).