Causal-Inspired Multitask Learning for Video-Based Human Pose Estimation

TL;DR

The paper tackles robustness and interpretability in video-based human pose estimation by adopting a causal perspective. It introduces CM-Pose, a two-stage framework comprising Multitask Coarse Learning (MCL) with self-supervised auxiliary tasks (masked token reconstruction and denoising) and Fine Token Enhancement (FTE) that distinguishes causal from non-causal tokens and clusters redundant non-causal tokens, all built on a criss-cross spatio-temporal attention backbone. The learning objective combines a pose heatmap loss with auxiliary-task reconstructions: $\\mathcal{L}_{total} = \\mathcal{L}_H + \lambda (\\mathcal{L}_{mask} + \\mathcal{L}_{denoise})$, while token-level mechanisms include similarity-based causal token selection $S^i = Softmax(\\frac{\\mathbf{Q}_k^i \\mathbf{K}_v^T}{\sqrt{D}})$ and density-peak clustering for non-causal tokens. CM-Pose achieves state-of-the-art results on PoseTrack2017/2018/2021 (e.g., $mAP=87.5$, $84.4$, $84.3$ respectively) and shows improved performance on occluded joints, demonstrating enhanced robustness and interpretability. The approach offers practical impact by enabling reliable pose estimation in challenging scenes without increasing inference cost, thanks to auxiliary tasks only during training and a token-based interpretability module at test time.

Abstract

Video-based human pose estimation has long been a fundamental yet challenging problem in computer vision. Previous studies focus on spatio-temporal modeling through the enhancement of architecture design and optimization strategies. However, they overlook the causal relationships in the joints, leading to models that may be overly tailored and thus estimate poorly to challenging scenes. Therefore, adequate causal reasoning capability, coupled with good interpretability of model, are both indispensable and prerequisite for achieving reliable results. In this paper, we pioneer a causal perspective on pose estimation and introduce a causal-inspired multitask learning framework, consisting of two stages. \textit{In the first stage}, we try to endow the model with causal spatio-temporal modeling ability by introducing two self-supervision auxiliary tasks. Specifically, these auxiliary tasks enable the network to infer challenging keypoints based on observed keypoint information, thereby imbuing causal reasoning capabilities into the model and making it robust to challenging scenes. \textit{In the second stage}, we argue that not all feature tokens contribute equally to pose estimation. Prioritizing causal (keypoint-relevant) tokens is crucial to achieve reliable results, which could improve the interpretability of the model. To this end, we propose a Token Causal Importance Selection module to identify the causal tokens and non-causal tokens (\textit{e.g.}, background and objects). Additionally, non-causal tokens could provide potentially beneficial cues but may be redundant. We further introduce a non-causal tokens clustering module to merge the similar non-causal tokens. Extensive experiments show that our method outperforms state-of-the-art methods on three large-scale benchmark datasets.

Figures (3)

• Figure 1: Overall pipeline of our CM-Pose framework. The goal is to identify the pose of keyframe $I_t$. CM-Pose includes two key components: Multitask Coarse Learning (MCL) and Fine Token Enhancement (FTE). (a) The MCL module consists of three tasks: the primary pose estimation task (middle branch), the masked token reconstruction task (upper branch), and the denoising token task (lower branch). To model the causal spatio-temporal dependencies for these tasks, we design a simple yet effective multitask learning network, a criss-cross spatio-temporal attention network. (b) The FTE module first performs temporal fusion on the coarse tokens. We then decouple the causal and non-causal tokens according to keypoint tokens attention. We further cluster non-causal tokens and merge the tokens from the same group into a new token. Finally, we concatenate the causal tokens and the merged non-causal tokens to estimate the pose $\mathbb{H}_t$ through a detection head.
• Figure 2: The keyframe (a) and visual comparisons of results obtained from FAMI (a), TDMI (b), and our CM-Pose (c) on challenging scenes in the PoseTrack dataset. Inaccurate predictions are highlighted with the red solid circles.
• Figure 3: Impact of different masked ratios and noise ratios in training process on PoseTrack2017 validation set.