Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video

TL;DR

This work tackles video-based human pose estimation by moving beyond pixel-wise motion cues to learn multi-level semantic motion dynamics and to densely fuse spatial and motion information. The authors propose MLSME to capture hierarchical inter-frame semantics through masked signal reconstruction and SMML to enable adaptive, cross-modal context sharing via cross-attention and learned pixel-wise fusion weights. Empirical results on PoseTrack2017, PoseTrack2018, and PoseTrack21 establish state-of-the-art performance and demonstrate the effectiveness of semantic dynamics and dense collaboration, especially under occlusion and blur. The approach offers a robust, efficient framework with potential extensions to 3D pose estimation and pose tracking in future work.

Abstract

Temporal modeling and spatio-temporal collaboration are pivotal techniques for video-based human pose estimation. Most state-of-the-art methods adopt optical flow or temporal difference, learning local visual content correspondence across frames at the pixel level, to capture motion dynamics. However, such a paradigm essentially relies on localized pixel-to-pixel similarity, which neglects the semantical correlations among frames and is vulnerable to image quality degradations (e.g. occlusions or blur). Moreover, existing approaches often combine motion and spatial (appearance) features via simple concatenation or summation, leading to practical challenges in fully leveraging these distinct modalities. In this paper, we present a novel framework that learns multi-level semantical dynamics and dense spatio-temporal collaboration for multi-frame human pose estimation. Specifically, we first design a Multi-Level Semantic Motion Encoder using a multi-masked context and pose reconstruction strategy. This strategy stimulates the model to explore multi-granularity spatiotemporal semantic relationships among frames by progressively masking the features of (patch) cubes and frames. We further introduce a Spatial-Motion Mutual Learning module which densely propagates and consolidates context information from spatial and motion features to enhance the capability of the model. Extensive experiments demonstrate that our approach sets new state-of-the-art results on three benchmark datasets, PoseTrack2017, PoseTrack2018, and PoseTrack21.

Figures (15)

• Figure 1: State-of-the-art methods like (a) TDMI feng2023mutual and (b) PoseWarper bertasius2019learning focus on modeling local pixel-wise dynamics based on feature similarities and they both show limitations during severe occlusion and blur. In contrast, our method (c) is more robust by fully exploiting the multi-level semantic motion contexts.
• Figure 2: Paradigm comparisons of existing pixel-wise motion estimation and our proposed multi-level semantical motion modeling.
• Figure 3: Overall pipeline of our proposed framework. The goal is to estimate the human pose in the keyframe. Given the input sequence, we first extract their spatial features using a visual encoder. The resulting feature tokens are then processed via two modules (b) MLSME and (c) SMML for motion feature extraction and spatial-motion feature aggregation. Finally, a detection head is employed to produce the final pose estimation $\hat{\mathbf{H}}_t^{i}$.
• Figure 4: Detailed structures of the sub-components, including (a) joint space-time attention, (b) divided space-time attention, and (c) Spatial-Motion Cross-Attention (SMCA).
• Figure 5: Visualization of temporal attention and space attention schemes. The patch in red indicates an arbitrary query patch within the input sequence, while blue and yellow patches represent the corresponding feature activations for temporal attention and space attention, respectively.