Lifelong Domain Adaptive 3D Human Pose Estimation

TL;DR

The paper tackles the challenge of generalizing 3D human pose estimations across non-stationary target pose distributions by introducing lifelong domain adaptive 3D HPE. It proposes a GAN-based framework with three 3D pose generators that encode pose-aware, temporal-aware, and domain-aware information, coupled with a 2D pose discriminator and a 2D-to-3D lifting estimator. A diffusion-based 2D pose sampler provides domain-aware priors to mitigate catastrophic forgetting, and an exponential moving average stabilizes updates across sequential target domains. Extensive experiments on H3.6M, MPI-INF-3DHP, and 3DPW demonstrate strong improvements over baselines in cross-scenario, cross-dataset, and multi-dataset lifelong adaptation, with ablations verifying the importance of domain-aware priors and EMA. The approach achieves practical gains with manageable computational overhead, highlighting its potential for real-world, continuously evolving 3D HPE deployments.

Abstract

3D Human Pose Estimation (3D HPE) is vital in various applications, from person re-identification and action recognition to virtual reality. However, the reliance on annotated 3D data collected in controlled environments poses challenges for generalization to diverse in-the-wild scenarios. Existing domain adaptation (DA) paradigms like general DA and source-free DA for 3D HPE overlook the issues of non-stationary target pose datasets. To address these challenges, we propose a novel task named lifelong domain adaptive 3D HPE. To our knowledge, we are the first to introduce the lifelong domain adaptation to the 3D HPE task. In this lifelong DA setting, the pose estimator is pretrained on the source domain and subsequently adapted to distinct target domains. Moreover, during adaptation to the current target domain, the pose estimator cannot access the source and all the previous target domains. The lifelong DA for 3D HPE involves overcoming challenges in adapting to current domain poses and preserving knowledge from previous domains, particularly combating catastrophic forgetting. We present an innovative Generative Adversarial Network (GAN) framework, which incorporates 3D pose generators, a 2D pose discriminator, and a 3D pose estimator. This framework effectively mitigates domain shifts and aligns original and augmented poses. Moreover, we construct a novel 3D pose generator paradigm, integrating pose-aware, temporal-aware, and domain-aware knowledge to enhance the current domain's adaptation and alleviate catastrophic forgetting on previous domains. Our method demonstrates superior performance through extensive experiments on diverse domain adaptive 3D HPE datasets.

Figures (10)

• Figure 1: (a) Comparisons among general domain adaptation generalDA, fine-tune finetune, source-free/test-time domain adaptation testtimeDA, and lifelong domain adaptation lifelongDA. (b) The paradigm of lifelong domain adaptive 3D human pose estimation.
• Figure 2: Overall adaption framework of our proposed lifelong domain adaptive 3D HPE approach at time $t = j$.
• Figure 3: Details of the unified paradigm for each of the three 3D pose generators in Fig. \ref{['fig:model']}. In contrast to existing generators, we introduce part segments (in purple color) for improved pose-aware encoding and introduce additional temporal-aware encoding (in blue color), leading to better adaptation on the current domain. Besides, we employ an unconditional 2D pose diffusion sampler to generate the domain-aware embedding (in cyan color), effectively mitigating catastrophic forgetting.
• Figure 4: Training (a) and sampling (b) processes of the 2D pose diffusion sampler.
• Figure 5: Visualization of H3.6M based on H3.6M: S1 $\rightarrow$ S5, S6, S7, S8 in Tab. \ref{['tab:h36m']}. The results are generated via the pose estimator obtained after $t = 4$. Left side is from S5 and right side is from S6.