Recurrent Feature Mining and Keypoint Mixup Padding for Category-Agnostic Pose Estimation

TL;DR

This work tackles category-agnostic pose estimation (CAPE) by probing fine-grained and structure-aware features from both support and query images. It introduces a recurrent FGSA feature mining framework built on deformable attention and guided by linked keypoint structures, together with a keypoint mixup padding strategy to unify the keypoint count across classes. The model refines support and query features across multiple layers, producing more accurate query keypoints as evidenced by substantial gains on the MP-100 dataset. The approach delivers richer supervision and improved generalization to novel classes, with code made available for reproducibility.

Abstract

Category-agnostic pose estimation aims to locate keypoints on query images according to a few annotated support images for arbitrary novel classes. Existing methods generally extract support features via heatmap pooling, and obtain interacted features from support and query via cross-attention. Hence, these works neglect to mine fine-grained and structure-aware (FGSA) features from both support and query images, which are crucial for pixel-level keypoint localization. To this end, we propose a novel yet concise framework, which recurrently mines FGSA features from both support and query images. Specifically, we design a FGSA mining module based on deformable attention mechanism. On the one hand, we mine fine-grained features by applying deformable attention head over multi-scale feature maps. On the other hand, we mine structure-aware features by offsetting the reference points of keypoints to their linked keypoints. By means of above module, we recurrently mine FGSA features from support and query images, and thus obtain better support features and query estimations. In addition, we propose to use mixup keypoints to pad various classes to a unified keypoint number, which could provide richer supervision than the zero padding used in existing works. We conduct extensive experiments and in-depth studies on large-scale MP-100 dataset, and outperform SOTA method dramatically (+3.2\%PCK@0.05). Code is avaiable at https://github.com/chenbys/FMMP.

Figures (5)

• Figure 1: Overview of representative methods and ours. (a): Representative methods use zero padding to align keypoint number, and rely on the heatmap pooling and cross-attention on single-scale features to produce support features and query estimations. (b): Our method uses mixup padding, and recurrently mines FGSA features from multi-scale features of support and query images, which can produce better support features and query estimations.
• Figure 2: The architecture of our $l$-th recurrent layer. Firstly, support keypoints $\bm{P}^*_s$ is aligned by mixup padding, which produces padded keypoints $\bm{P}_s$. Then, $f_{miner-s}$ mines FGSA features around $\bm{P}_s$ on support image to extract support features $\bm{F}^l_s$ according to the last keypoint features $\bm{F}^{l-1}_q$. Afterwards, $f_{miner-q}$ mines FGSA features around the last keypoints $\bm{P}^{l-1}_q$ on query image to extract keypoints $\bm{F}^l_q$, which produces target keypoints $\bm{P}^l_q$. By recurrently updating $\bm{F}^l_s$, $\bm{F}^l_q$ and $\bm{P}^l_q$, our framework finally estimates precise keypoints $\bm{P}^L_q$.
• Figure 3: Illustration of keypoint mixup padding, i.e., padding $16$ support keypoints (upper) and target keypoints (bottom) to $35$ or $70$ keypoints. Therefore, our padding strategy could provide richer superivsion and enbale model to learn denser semantics.
• Figure 4: Qualitative comparison. The left two columns show the GT support and target keypoints on support and query images. The right four columns show the target keypoints on query images estimated by various methods. We employ graduated colors to denote the semantic classes of keypoints, and use small black arrows to indicate the deviations to GT target keypoints (i.e., smaller arrows are better).
• Figure 5: Visulization for the attention on support image (col-1) and query (col-2) image, where the red cross indicates the reference point, and the green point indicates attention point with transparency as weight. The green circle in col-3 shows estimated keypoint and the red arrow shows the deviation to GT.