Quantifying Uncertainty in Machine Learning-Based Pervasive Systems: Application to Human Activity Recognition

TL;DR

The study tackles unreliable predictions from ML-based pervasive HAR systems by proposing a cascaded uncertainty quantification framework that combines a MAE-based Transformer, reconstruction-loss thresholds, latent-space distance, and MC dropout. It demonstrates that different uncertainty cues excel under distinct distributional shifts and that cascading these cues improves detection of out-of-distribution inputs while remaining computationally feasible on edge devices. The approach is validated on RealWorld and HHAR datasets, using a green/red flag scheme to communicate reliability to domain operators. This work advances practical, risk-aware deployment of HAR in IoT environments and outlines avenues for further enhancement with density-based and evidential learning techniques.

Abstract

The recent convergence of pervasive computing and machine learning has given rise to numerous services, impacting almost all areas of economic and social activity. However, the use of AI techniques precludes certain standard software development practices, which emphasize rigorous testing to ensure the elimination of all bugs and adherence to well-defined specifications. ML models are trained on numerous high-dimensional examples rather than being manually coded. Consequently, the boundaries of their operating range are uncertain, and they cannot guarantee absolute error-free performance. In this paper, we propose to quantify uncertainty in ML-based systems. To achieve this, we propose to adapt and jointly utilize a set of selected techniques to evaluate the relevance of model predictions at runtime. We apply and evaluate these proposals in the highly heterogeneous and evolving domain of Human Activity Recognition (HAR). The results presented demonstrate the relevance of the approach, and we discuss in detail the assistance provided to domain experts.

Figures (7)

• Figure 1: Cross-dataset evaluation with popular datasets. A cell at coordinates $i,j$ shows the F1 score obtained by training the model with $70\%$ of dataset $i$ on $30\%$ of dataset $j$presotto2023.
• Figure 2: Overall approach
• Figure 3: Model architecture combining MAE with a HAR Transformer Ek2024.
• Figure 4: Reconstruction loss distribution for the training (blue) and validation (red) datasets, with the red line indicating the 0.99 quantile in the validation set.
• Figure 5: Distribution of distances to the nearest clusters for the training (blue) and validation (orange) datasets, with the red line indicating the 99% quantile in the validation set.