RhythmFormer: Extracting Patterned rPPG Signals based on Periodic Sparse Attention

TL;DR

RhythmFormer addresses the challenge of extracting reliable rPPG signals from facial videos by exploiting the quasi-periodic nature of PPG through a novel periodic sparse attention mechanism guided by a pre-attention stage. A plug-and-play fusion stem enhances frame-level rPPG cues and steers self-attention toward weak BVP signals, enabling finer-grained feature extraction with reduced computation. The method stacks a hierarchical temporal periodic transformer with three TPT blocks and a multi-scale temporal fusion, achieving state-of-the-art results on five public datasets and transferring well across datasets. The work provides a practical, open-source baseline for rPPG with potential extension to multimodal periodic video understanding.

Abstract

Remote photoplethysmography (rPPG) is a non-contact method for detecting physiological signals based on facial videos, holding high potential in various applications. Due to the periodicity nature of rPPG signals, the long-range dependency capturing capacity of the transformer was assumed to be advantageous for such signals. However, existing methods have not conclusively demonstrated the superior performance of transformers over traditional convolutional neural networks. This may be attributed to the quadratic scaling exhibited by transformer with sequence length, resulting in coarse-grained feature extraction, which in turn affects robustness and generalization. To address that, this paper proposes a periodic sparse attention mechanism based on temporal attention sparsity induced by periodicity. A pre-attention stage is introduced before the conventional attention mechanism. This stage learns periodic patterns to filter out a large number of irrelevant attention computations, thus enabling fine-grained feature extraction. Moreover, to address the issue of fine-grained features being more susceptible to noise interference, a fusion stem is proposed to effectively guide self-attention towards rPPG features. It can be easily integrated into existing methods to enhance their performance. Extensive experiments show that the proposed method achieves state-of-the-art performance in both intra-dataset and cross-dataset evaluations. The codes are available at https://github.com/zizheng-guo/RhythmFormer.

Figures (13)

• Figure 1: Sparsity in attention due to periodicity. Given the red block as a Query, the global spatio-temporal correlations are computed. Brighter regions in the attention score (top row) and key response (bottom row) in the frame indicate stronger attention. From the figure, it is evident that the brightness of the key response in the frame is predominantly concentrated in the facial area, while the brightness in the attention score is mainly focused on the troughs (regions with the same phase as the Query). The attention scores exhibit a notable sparsity.
• Figure 2: The proposed periodic sparse attention mechanism utilizes the periodic patterns learned through pre-attention to gather high-score regions (facial regions with similar temporal phases), which filters out a substantial number of irrelevant attention computations. This allows for finer-grained token extraction across dimensions H, W, and T compared with vanilla attention.
• Figure 3: The general framework of RhythmFormer. It consists of fusion stem, patch embedding, hierarchical temporal periodic transformer, and rPPG predictor head. The hierarchical temporal periodic transformer consists of three temporal periodic transformer (TPT) Blocks and two temporal downsampling modules between TPT Blocks. The attention mechanism in the TPT block is periodic sparse attention. It introduces a pre-attention stage before the vanilla attention, which selects the top-k tokens to filter out irrelevant key (K) and value (V).
• Figure 4: Visualization comparison of features extracted by the fusion stem and frame differences.
• Figure 5: Heart rate distribution among different datasets.