SigFormer: Sparse Signal-Guided Transformer for Multi-Modal Human Action Segmentation

TL;DR

SigFormer addresses multi-modal human action segmentation by leveraging sparse IoT sensor signals together with dense modalities through a masked-attention-based sparse guided fusion. It introduces an intermediate bottleneck to enforce action-class and temporal-boundary awareness per modality and employs mutual interactive branches to explicitly model the interplay between category predictions and boundaries. The framework achieves state-of-the-art performance on the OpenPack industrial dataset, with a macro F1 score of $0.958$ when fusing four modalities, while providing robust results across individual modalities. This approach enhances temporal localization and reduces over-segmentation, offering practical benefits for real-world industrial human-action understanding and IoT-assisted perception.

Abstract

Multi-modal human action segmentation is a critical and challenging task with a wide range of applications. Nowadays, the majority of approaches concentrate on the fusion of dense signals (i.e., RGB, optical flow, and depth maps). However, the potential contributions of sparse IoT sensor signals, which can be crucial for achieving accurate recognition, have not been fully explored. To make up for this, we introduce a Sparse signalguided Transformer (SigFormer) to combine both dense and sparse signals. We employ mask attention to fuse localized features by constraining cross-attention within the regions where sparse signals are valid. However, since sparse signals are discrete, they lack sufficient information about the temporal action boundaries. Therefore, in SigFormer, we propose to emphasize the boundary information at two stages to alleviate this problem. In the first feature extraction stage, we introduce an intermediate bottleneck module to jointly learn both category and boundary features of each dense modality through the inner loss functions. After the fusion of dense modalities and sparse signals, we then devise a two-branch architecture that explicitly models the interrelationship between action category and temporal boundary. Experimental results demonstrate that SigFormer outperforms the state-of-the-art approaches on a multi-modal action segmentation dataset from real industrial environments, reaching an outstanding F1 score of 0.958. The codes and pre-trained models have been available at https://github.com/LIUQI-creat/SigFormer.

Figures (6)

• Figure 1: Overview of the pipeline for SigFormer. (a) Feature extraction from each signal. (b) The intermediate bottleneck module operates by taking $F_i$ ($i$$\in M, K, B$) as input, reducing its feature dimension, calculating the inner class loss $\mathcal{L}_{c}^{inner}$, and the inner boundary loss $\mathcal{L}_{b}^{inner}$. Subsequently, it restores the feature dimension to obtain $F_o$ ($o$$\in M, K, B$), which possesses an enhanced awareness of both category and boundary. (c) The sparse guided cross-modal module combines features from multi-modal signals with different information densities, $F_M$, $F_D$, and $F_S$ (obtained by adding $F_K$ and $F_B$), to generate a comprehensive action representation $F_C$. This module comprises a sparse guided fusion (SGF) module and a motion-spatial attention fusion (MSAF) module. (d) Mutual interactive branches employ Multi-Head Cross Attention (MHCA) to model information interaction between the class branch and boundary branch. The predictions of the class branch are refined by the boundaries (filtered from the predictions of the boundary branch) to obtain frame-level refined class predictions.
• Figure 2: The architecture of the sparse guided cross-modal module. It is formed by two parts: sparse guided fusion (SGF) module and motion-spatial attention fusion (MSAF) module. The SGF module utilizes Multi-Head Masked Attention (MHMA) to concentrate attention on temporal regions with active sparse signals, ensuring their accurate guidance of other signals. The Multi-Head Cross Attention (MHCA) in the MSAF is employed to capture the interaction context between motion and spatial information.
• Figure 3: Qualitative results on OpenPack. SigFormer effectively utilizes multi-modal information and alleviates the issue of over-segmentation more efficiently compared to other methods.
• Figure 4: Accuracy on each action class for each modality and combination of them.
• Figure 5: The prediction results obtained from models employing various fusion methods. (a) Concatenate multi-modal features. (b) BF blocks leverage the high-confidence sparse signals as anchors and fuse them with dense sensor signals. (c) The mask attention in the SGF module is replaced by normal cross-attention to demonstrate its effectiveness. (d) Ours. The figures display the time-series data of the ground truth (red line), along with the class estimates represented as probabilities. High- and low-class probabilities are yellow and blue, respectively. SigFormer's predictions closely align with the ground truth, effectively distinguishing similar actions and avoiding unstable estimates.