Hi-OSCAR: Hierarchical Open-set Classifier for Human Activity Recognition

TL;DR

This work addresses the gap in HAR for unseen activities by introducing Hi-OSCAR, a Hierarchical Open-set Classifier that uses a self-supervised, ResNet-based feature extractor and HAC-generated hierarchies to enable accurate ID classification while localizing and signaling OOD samples. By training with L_ID and L_OOD losses and employing an inference stopping criterion, the method yields informative internal-node predictions for near-OOD activities and robust OOD detection via mean path entropy. The authors also introduce NFI_FARED, a richly annotated HAR dataset with diverse contexts and ample samples to assess open-set performance. Overall, Hi-OSCAR achieves strong ID accuracy and OOD detection across multiple datasets, including NFI_FARED, and demonstrates that hierarchical structure and internal-node localization can improve interpretability and reliability in HAR systems.

Abstract

Within Human Activity Recognition (HAR), there is an insurmountable gap between the range of activities performed in life and those that can be captured in an annotated sensor dataset used in training. Failure to properly handle unseen activities seriously undermines any HAR classifier's reliability. Additionally within HAR, not all classes are equally dissimilar, some significantly overlap or encompass other sub-activities. Based on these observations, we arrange activity classes into a structured hierarchy. From there, we propose Hi-OSCAR: a Hierarchical Open-set Classifier for Activity Recognition, that can identify known activities at state-of-the-art accuracy while simultaneously rejecting unknown activities. This not only enables open-set classification, but also allows for unknown classes to be localized to the nearest internal node, providing insight beyond a binary "known/unknown" classification. To facilitate this and future open-set HAR research, we collected a new dataset: NFI_FARED. NFI_FARED contains data from multiple subjects performing nineteen activities from a range of contexts, including daily living, commuting, and rapid movements, which is fully public and available for download.

Figures (8)

• Figure 1: Model inference for an Out-of-Distribution activity using Hi-OSCAR. Raw sensor data are separated sensor-wise and fed through a feature extractor, giving an output vector of length $|H|$ (where $|H|$ > number of classes). Elements of the output vector correspond with nodes in the hierarchy. The Path Likelihood of a class is computed by multiplying the softmax scores of each node along the path from root to node, and highest path likelihood determines the predicted path. If the decision entropy of any node along the chosen path exceeds the selected threshold, prediction is terminated. As an example here, the OOD class kicking would classified as cycling in the closed-set system, but due to high entropy it is classed as OOD. Prediction is also terminated at the penultimate node due to high decision entropy, indicating although it is OOD, it is similar to cycling and running.
• Figure 2: Feature extractor architecture adapted from Yuan et al. yuan_self-supervised_2022, extended to the multi-sensor, multi-device domain.
• Figure 3: Dendrogram hierarchy generated using Hierarchical Agglomerative Clustering on NFI_FARED with total Cosine Distance after each merge indicated on the y-axis.
• Figure 4: Illustrative example of training and inference on the Hi-OSCAR hierarchy. (a) Visualisation of optimal activations during training. Nodes along the path to correct ID class are optimised by $L_{ID}$, while nodes off the path are optimised to the uniform distribution by $L_{OOD}$ (b) Example visualisation of inference on OOD class elevator, showing example node activations. Standing has the highest path probability, however due to high entropy in the final level, an internal node is outputted.
• Figure 5: Number of 10 second samples from each dataset. Blank squares denote the activities not in that dataset.