Domain Generalization for Improved Human Activity Recognition in Office Space Videos Using Adaptive Pre-processing

TL;DR

This work tackles domain generalization for office-space human activity recognition in FPV data by introducing encoder-agnostic, pre-processing techniques that generate dual frame stacks (raw and YOLOv7-masked), plus an adaptive, single-stream selection mechanism guided by a Learnable Decision Embedding and a Class Array, and a Framewise Attention module. The approach yields a dynamic, object-aware input strategy and frame-level weighting that improve generalization to unseen domains, demonstrated on the BON FPV dataset with different backbones, notably MViT, outperforming domain adaptation baselines in both seen and unseen domains. Ablation studies confirm the contributions of the decision embedding and framewise attention, and empirical results show significant gains in accuracy, precision, recall, and F1 scores for unseen domains. The method offers a practical, plug-and-play preprocessing framework to boost cross-domain HAR in realistic office environments, with potential extension to broader domains and modalities.

Abstract

Automatic video activity recognition is crucial across numerous domains like surveillance, healthcare, and robotics. However, recognizing human activities from video data becomes challenging when training and test data stem from diverse domains. Domain generalization, adapting to unforeseen domains, is thus essential. This paper focuses on office activity recognition amidst environmental variability. We propose three pre-processing techniques applicable to any video encoder, enhancing robustness against environmental variations. Our study showcases the efficacy of MViT, a leading state-of-the-art video classification model, and other video encoders combined with our techniques, outperforming state-of-the-art domain adaptation methods. Our approach significantly boosts accuracy, precision, recall and F1 score on unseen domains, emphasizing its adaptability in real-world scenarios with diverse video data sources. This method lays a foundation for more reliable video activity recognition systems across heterogeneous data domains.

Figures (8)

• Figure 1: Differences caused by domain shift. Due to the difference in location, the office environment along with the point of view and person's skin color is completely different even for the same activity category.
• Figure 2: Evaluation of performance degradation of video classifiers due to domain disparity.
• Figure 3: Proposed Preprocessing Framework. In the input to the framework we are giving the Frame Tensor ($\mathbf{\hat{F}}$), Class Array ($\mathbf{\hat{C}}$), and the Video Segment and at the output we are having the activity class happening in the video segment.
• Figure 4: Frame extracted from video clips are sub-sampled to get 'Raw frame stack', which is then passed through YOLLOv7 algorithm to generate 'Masked frame stack'.
• Figure 5: (a) Adaptive stream selection block: The c-dimensional vector class array $\hat{C}$ and the trainable weight vector $W$ are multiplied to generate a scalar $D_2$. Which is subtracted from $D_1$, obtained from the MLP network. The resulting $D$ is processed through a ReLU decision block to produce the output $\hat{D}$, which is either zero or positive. (b) Framewise attention mechanism: The frame tensor $\mathbf{\hat{F}}$ (of size $(n,c)$) is passed through the same MLP network to generate an attention vector $\mathbf{A}$.