Detecting Informative Channels: ActionFormer

TL;DR

This work adapts the Transformer-based ActionFormer from video TAL to inertial HAR by introducing an adaptive channel-wise enhancement that leverages squeeze-and-excitation and the Swish activation to correctly weigh channel-wise information. It also proposes a MaxPool variant for video inputs and cross-layer connections to improve feature fusion, achieving a substantial improvement in average mAP on the WEAR inertial dataset ($ ext{Avg mAP} = 73.27$ at CL=$0.5$, $tIoU \in [0.3,0.7]$) over the baseline. Ablation studies show the SE module and adaptive channel weighting as primary contributors to gains, with manageable parameter and runtime overhead. The approach demonstrates strong, robust HAR performance with inertial data and generalizes to video features, offering practical impact for wearable sensing, real-time activity detection, and multimodal HAR systems, while outlining concrete future directions in feature extraction, data augmentation, and cross-modal integration.

Abstract

Human Activity Recognition (HAR) has recently witnessed advancements with Transformer-based models. Especially, ActionFormer shows us a new perspectives for HAR in the sense that this approach gives us additional outputs which detect the border of the activities as well as the activity labels. ActionFormer was originally proposed with its input as image/video. However, this was converted to with its input as sensor signals as well. We analyze this extensively in terms of deep learning architectures. Based on the report of high temporal dynamics which limits the model's ability to capture subtle changes effectively and of the interdependencies between the spatial and temporal features. We propose the modified ActionFormer which will decrease these defects for sensor signals. The key to our approach lies in accordance with the Sequence-and-Excitation strategy to minimize the increase in additional parameters and opt for the swish activation function to retain the information about direction in the negative range. Experiments on the WEAR dataset show that our method achieves substantial improvement of a 16.01\% in terms of average mAP for inertial data.

Figures (11)

• Figure 1: Architectural comparison of conventional HAR frameworks and our proposed approaches. (a) Conventional method: CNNs. (b) ActionFormer architecture: An enhanced framework that employs Transformer-based layers as the primary backbone components. (c) Adapter method (Standard Adapter and TIA methods): Use adapter after basic layer to strengthen the framework. (d) Our proposed framework: Integrates dedicated modules following each basic layer to strengthen feature representations. (e) Alternative module placement strategy: An optimized configuration designed to minimize information loss while reducing computational complexity and memory requirements.
• Figure 2: (a)Adaptive Channel-wise Enhancement Module tailored for inertial data versus; (b) MaxPool Chanel-wise Enhancement Module tailored for visual data
• Figure 3: Overview of our backbone. We build our refined encoder (illustrated within the green box on the right side of the figure), which can direct the model's attention toward important channels, thereby improving the ability to extract spatial and temporal features.
• Figure 4: Confusion matrix of original model being applied using inertial data.
• Figure 5: Confusion matrix of our model being applied using inertial data.