SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos

TL;DR

STDPose tackles the challenge of human pose estimation in sparsely labeled videos by learning spatiotemporal dynamics and leveraging the interaction between pose heatmaps and visual features. It introduces a SpatioTemporal Representation Encoder (STRE), a Dynamic-Aware Mask (DAM), and a SpatioTemporal Dynamics Aggregation (STDA), guided by a mutual information objective to selectively fuse temporal cues. The method achieves state-of-the-art results on PoseTrack2017/2018/2021 for both pose propagation and pose estimation, including strong data-efficiency with pseudo-labels that enable competitive performance with only a fraction of labeled frames. By reducing annotation requirements and improving robustness to occlusion and blur, STDPose offers a practical, scalable solution for video-based human pose analysis.

Abstract

Human pose estimation in videos remains a challenge, largely due to the reliance on extensive manual annotation of large datasets, which is expensive and labor-intensive. Furthermore, existing approaches often struggle to capture long-range temporal dependencies and overlook the complementary relationship between temporal pose heatmaps and visual features. To address these limitations, we introduce STDPose, a novel framework that enhances human pose estimation by learning spatiotemporal dynamics in sparsely-labeled videos. STDPose incorporates two key innovations: 1) A novel Dynamic-Aware Mask to capture long-range motion context, allowing for a nuanced understanding of pose changes. 2) A system for encoding and aggregating spatiotemporal representations and motion dynamics to effectively model spatiotemporal relationships, improving the accuracy and robustness of pose estimation. STDPose establishes a new performance benchmark for both video pose propagation (i.e., propagating pose annotations from labeled frames to unlabeled frames) and pose estimation tasks, across three large-scale evaluation datasets. Additionally, utilizing pseudo-labels generated by pose propagation, STDPose achieves competitive performance with only 26.7% labeled data.

Figures (5)

• Figure 1: Our model, STDPose, consistently demonstrates high accuracy and robustness in human pose estimation, even in challenging video scenes with blur and occlusion, thanks to its innovative approach to capturing spatiotemporal information and long-range motion cues. However, state-of-the-art methods like PoseWarper bertasius2019learning and DCPose liu2021deep struggle in such scenarios. Red rectangles in our visual comparisons indicate where these methods completely failed to detect blurred individuals, while red ellipses highlight their incorrect detections of wrist and ankle joints caused by severe occlusion. All results are from models trained on sparsely labeled (i.e., every 7 frames) videos.
• Figure 2: The overall pipeline of our STDPose framework. Given an input sequence $\left\langle I_{l}^{i}, I_{t}^{i} , I_{r}^{i}\right\rangle$, our goal is to estimate the human pose of the key frame $I_{t}^{i}$.
• Figure 3: Visual results of our STDPose on the PoseTrack2017 iqbal2017posetrack dataset include challenging scenes, such as rapid movements and pose occlusions.
• Figure 4: Visual results of our STDPose on the PoseTrack2018 andriluka2018posetrack dataset include challenging scenes, such as rapid movements and pose occlusions.
• Figure 5: Visual results of our STDPose on the PoseTrack2021 doering2022posetrack21 dataset include challenging scenes, such as rapid movements and pose occlusions.