HAR-DoReMi: Optimizing Data Mixture for Self-Supervised Human Activity Recognition Across Heterogeneous IMU Datasets

TL;DR

The paper tackles the challenge of cross-dataset generalization in HAR by reframing pre-training data composition through HAR-DoReMi, a data-mixture optimization framework inspired by DoReMi and Group DRO. It adapts DoReMi to HAR via a masked reconstruction objective and integrates the Mahony pose fusion to align heterogeneous IMU sensor orientations, significantly improving cross-dataset performance with reduced data needs. The authors demonstrate an average accuracy improvement of approximately $6.51\%$ on four public HAR datasets using only about $30\%$ to $50\%$ of the data, and show strong data efficiency and robustness across multi-sensor setups. These results highlight the practical potential of HAR-DoReMi for scalable, generalizable HAR pre-training in real-world, heterogeneous sensor environments.

Abstract

Cross-dataset Human Activity Recognition (HAR) suffers from limited model generalization, hindering its practical deployment. To address this critical challenge, inspired by the success of DoReMi in Large Language Models (LLMs), we introduce a data mixture optimization strategy for pre-training HAR models, aiming to improve the recognition performance across heterogeneous datasets. However, directly applying DoReMi to the HAR field encounters new challenges due to the continuous, multi-channel and intrinsic heterogeneous characteristics of IMU sensor data. To overcome these limitations, we propose a novel framework HAR-DoReMi, which introduces a masked reconstruction task based on Mean Squared Error (MSE) loss. By raplacing the discrete language sequence prediction task, which relies on the Negative Log-Likelihood (NLL) loss, in the original DoReMi framework, the proposed framework is inherently more appropriate for handling the continuous and multi-channel characteristics of IMU data. In addition, HAR-DoReMi integrates the Mahony fusion algorithm into the self-supervised HAR pre-training, aiming to mitigate the heterogeneity of varying sensor orientation. This is achieved by estimating the sensor orientation within each dataset and facilitating alignment with a unified coordinate system, thereby improving the cross-dataset generalization ability of the HAR model. Experimental evaluation on multiple cross-dataset HAR transfer tasks demonstrates that HAR-DoReMi improves the accuracy by an average of 6.51%, compared to the current state-of-the-art method with only approximately 30% to 50% of the data usage. These results confirm the effectiveness of HAR-DoReMi in improving the generalization and data efficiency of pre-training HAR models, underscoring its significant potential to facilitate the practical deployment of HAR technology.

Figures (5)

• Figure 1: DoReMi optimizes data mixture for large language model in three steps: (Step 1) Train a reference model with initial domain weights. (Step 2) Train a proxy model using the Group DRO algorithm to optimize domain weights. (Step 3) Train a large language model using the dataset reweighted by the optimized weights. This method effectively tunes data proportions for improving model performance.
• Figure 2: HAR-DoReMi Framework Workflow. For scenarios without specified target datasets, the workflow comprises four steps: (Step 1) Data pre-processing via the Mahony algorithm is fed as input to model. (Step 2) A reference model is trained to establish baseline loss across each domain. (Step 3) Next, a proxy model is trained employing the Group DRO algorithm to minimize excess domain loss relative to the reference model, with outputting the average domain weights obtained by the training of proxy model. (Step 4) Lastly, training data is aggregated based on the average domain weights, and this combined data serves as input for the training of final target model.
• Figure 3: Comparison of Domain Weights Obtained by HAR-DoReMi with and without Mahony Algorithm. (a) The domain weights derived from HAR-DoReMi without utilizing the Mahony algorithm for data transformation, reflect the initial importance and influence of the original dataset during training. (b)The domain weights obtained by HAR-DoReMi with the integration of the Mahony algorithm for data transformation, demonstrate the influence of Mahony algorithm on the weight distribution of the dataset during training. The results in the figure are all obtained at 1000 steps of training.
• Figure 4: Domain Weights Comparison for Shoaib Dataset Sensor Placement Subsets. The figure presents domain weights from HAR-DoReMi training on Shoaib dataset subsets (partitioned by sensor location) at 1000 training steps. (a) Domain weights without Mahony algorithm data transformation. (b) Domain weights with Mahony algorithm data transformation.
• Figure 5: Comparison Figure of the Mahony Algorithm's Alignment Effect on IMU Data for Identical Activities in the HAR Dataset. The upper half of the figure illustrates the application effect of the Mahony algorithm on the “Downstairs” activity. The left side shows the raw data, while the right side presents the data processed by the Mahony algorithm. The lower half demonstrates the effect of the Mahony algorithm on the “Still” activity. Similarly, the left side displays the raw data, and the right side shows the processed data.