Incremental Object Keypoint Learning

TL;DR

Incremental Object Keypoint Learning (IKL) tackles updating a keypoint estimator with data labeled only for newly defined keypoints while not retaining old data. The proposed KAMP baseline uses a two-stage strategy: Stage I Knowledge Association (KA) trains KA-Net to model anatomical relations between old and new keypoints, and Stage II Mutual Promotion (MP) jointly refines old and new keypoints via a combination of supervised learning and a novel spatial distillation loss that operates over heatmap spatial dimensions. Empirical results across Head-2023, Chest, MPII, and ATRW demonstrate reduced forgetting and even positive transfer for old keypoints, with strong labeling efficiency in low-shot regimes. These findings show that explicitly modeling inter-keypoint relations yields practical gains for IKL, enabling scalable, exemplar-free incremental keypoint learning in diverse domains.

Abstract

Existing progress in object keypoint estimation primarily benefits from the conventional supervised learning paradigm based on numerous data labeled with pre-defined keypoints. However, these well-trained models can hardly detect the undefined new keypoints in test time, which largely hinders their feasibility for diverse downstream tasks. To handle this, various solutions are explored but still suffer from either limited generalizability or transferability. Therefore, in this paper, we explore a novel keypoint learning paradigm in that we only annotate new keypoints in the new data and incrementally train the model, without retaining any old data, called Incremental object Keypoint Learning (IKL). A two-stage learning scheme as a novel baseline tailored to IKL is developed. In the first Knowledge Association stage, given the data labeled with only new keypoints, an auxiliary KA-Net is trained to automatically associate the old keypoints to these new ones based on their spatial and intrinsic anatomical relations. In the second Mutual Promotion stage, based on a keypoint-oriented spatial distillation loss, we jointly leverage the auxiliary KA-Net and the old model for knowledge consolidation to mutually promote the estimation of all old and new keypoints. Owing to the investigation of the correlations between new and old keypoints, our proposed method can not just effectively mitigate the catastrophic forgetting of old keypoints, but may even further improve the estimation of the old ones and achieve a positive transfer beyond anti-forgetting. Such an observation has been solidly verified by extensive experiments on different keypoint datasets, where our method exhibits superiority in alleviating the forgetting issue and boosting performance while enjoying labeling efficiency even under the low-shot data regime.

Figures (9)

• Figure 1: Medical analysis can frequently change and require new keypoints essentially Cao2023-wf, while the labeling is time-consuming.
• Figure 2: Separate Estimators need multiple estimators in test time. Unsupervised keypoint learning (UKL) and category-agnostic pose estimation (CAPE) exploit a fixed pretrained keypoint estimator on unseen new keypoints. While our Incremental object Keypoint Kearning (IKL) continually updates the same model on the new data labeled only on new keypoints without retaining any old training data.
• Figure 3: Overview of KAMP using the human body for illustration. In Stage-I, we learn an auxiliary KA-Net to associate the related old and new keypoints based on their local anatomical constraint. In Stage-II, we jointly leverage the old model and the KA-Net as an auxiliary teacher to consolidate all old keypoints' prediction and also learn the new keypoints simultaneously to achieve mutual promotions.
• Figure 4: Qualitative results on Split Head-2023 and MPII. All methods start from the same Step-0 model, whose prediction is in the second column. GT: ground truth. The red circles denote the keypoints learned in Step 0, and the green ones denote all the new keypoints learned in later incremental steps. We observe that after IKL, the compared methods (LWF and CPR) may acquire the new keypoints as ours, but they have obvious miss-detection and wrong estimation (e.g., out of the body). While our method can consistently associate the new and old keypoints and achieve structurally accurate keypoint predictions. More results are included in our Supplementary.
• Figure 5: Overview of KAMP using the human body for illustration. In Stage-I, we learn an auxiliary KA-Net to associate the related old and new keypoints based on their local anatomical constraint. In Stage-II, we jointly leverage the old model and the KA-Net as an auxiliary teacher to consolidate all old keypoints' prediction and also learn the new keypoints simultaneously to achieve mutual promotions.