Optimizing Human Pose Estimation Through Focused Human and Joint Regions

TL;DR

The paper addresses the challenge of video-based human pose estimation where global pixel-wise processing distracts from the target body and joints. It introduces VREMD, a dual-stream Transformer framework that combines a Human-Keypoint Mask Enhanced (HKME) module for coarse-to-fine region focus and a Bidirectional Motion Disentanglement (BMD) module with a Deformable Cross Attention (DCA) constraint to extract pose-related motion from a constrained tri-frame context $\{I_{t-1}^i, I_t^i, I_{t+1}^i\}$. Key contributions include the HKME dual-mask mechanism, the Adaptive Deformable Cross block with bidirectional motion separation, and the demonstration of state-of-the-art results on PoseTrack2017/2018/2021, including a wrist $mAP$ of $84.8$ on PoseTrack2017. The work advances practical, region-focused Transformer-based pose estimation in videos, with potential impact on applications requiring robust local localization under occlusion or motion blur.

Abstract

Human pose estimation has given rise to a broad spectrum of novel and compelling applications, including action recognition, sports analysis, as well as surveillance. However, accurate video pose estimation remains an open challenge. One aspect that has been overlooked so far is that existing methods learn motion clues from all pixels rather than focusing on the target human body, making them easily misled and disrupted by unimportant information such as background changes or movements of other people. Additionally, while the current Transformer-based pose estimation methods has demonstrated impressive performance with global modeling, they struggle with local context perception and precise positional identification. In this paper, we try to tackle these challenges from three aspects: (1) We propose a bilayer Human-Keypoint Mask module that performs coarse-to-fine visual token refinement, which gradually zooms in on the target human body and keypoints while masking out unimportant figure regions. (2) We further introduce a novel deformable cross attention mechanism and a bidirectional separation strategy to adaptively aggregate spatial and temporal motion clues from constrained surrounding contexts. (3) We mathematically formulate the deformable cross attention, constraining that the model focuses solely on the regions centered at the target person body. Empirically, our method achieves state-of-the-art performance on three large-scale benchmark datasets. A remarkable highlight is that our method achieves an 84.8 mean Average Precision (mAP) on the challenging wrist joint, which significantly outperforms the 81.5 mAP achieved by the current state-of-the-art method on the PoseTrack2017 dataset.

Figures (4)

• Figure 1: A high-level overview of our proposed VREMD, which utilizes a dual-stream architecture to collaboratively process and integrate complementary visual and motion features. The visual representation stream executes progressive enhancement of human keypoint-related features to achieve precise location recognition. The motion stream performs adaptive pose-related motion disentanglement through the novel deformable cross attention. $\{\boldsymbol{F}_{t-1}^{i}, \boldsymbol{F}_{t}^{i}, \boldsymbol{F}_{t+1}^{i}\}$ denote the visual features of three input frames $\{I_{t-1}^{i}, I_{t}^{i}, I_{t+1}^{i}\}$ output by backbone network.
• Figure 2: The overall pipeline of our VREMD framework. Given an input sequence $\{ I_{t-1}^{i}, I_{t}^{i} , I_{t+1}^{i}\}$, our goal is to estimate the human pose of the key frame $I_{t}^{i}$. We initially extract the visual features via a ViT backbone, and then feed them into the Human-Keypoint Enhanced module and the Bidirectional Motion Disentanglement module to obtain $\boldsymbol{T}_{t}^{i}$ and $\boldsymbol{M}$. Finally, the outputs derived from different heads are combined through a weighted sum to arrive at the final predicted pose heatmap $\boldsymbol{H}_{t}^{i}$.
• Figure 3: Qualitative comparison of our VREMD, DCPose liu2021dcpose, and TDMI feng2023tdmi on the PoseTrack2017 dataset, featuring challenges such as pose occlusions, fast motion, and video defocus. Red solid circles denote the inaccurate pose predictions.
• Figure 4: Visual results of our VREMD on PoseTrack datasets. Challenging Scenarios such as fast motion or pose occlusion are involved.