L3D-Pose: Lifting Pose for 3D Avatars from a Single Camera in the Wild

TL;DR

The work tackles the gap between 2D pose estimation and true 3D understanding in animals by leveraging rigged avatars and Unity-generated synthetic data to train an image-independent Attention MLP that lifts 2D keypoints to a partial 3D pose. A Look-up Table maps these soft 3D poses to Deep 3D poses, enabling retargeting and texture transfer onto avatars, and two synthetic datasets, Deep Macaque and Deep Horse, are introduced to support this framework. The approach achieves accurate 3D pose reconstruction and avatar retargeting without real-world 3D annotations, demonstrating strong generalization to wild images and practical applicability for graphics and behavioral analysis. Overall, L3D-Pose provides a scalable, data-efficient pathway for lifting 2D poses to 3D and retargeting them to arbitrary avatars in uncontrolled environments.

Abstract

While 2D pose estimation has advanced our ability to interpret body movements in animals and primates, it is limited by the lack of depth information, constraining its application range. 3D pose estimation provides a more comprehensive solution by incorporating spatial depth, yet creating extensive 3D pose datasets for animals is challenging due to their dynamic and unpredictable behaviours in natural settings. To address this, we propose a hybrid approach that utilizes rigged avatars and the pipeline to generate synthetic datasets to acquire the necessary 3D annotations for training. Our method introduces a simple attention-based MLP network for converting 2D poses to 3D, designed to be independent of the input image to ensure scalability for poses in natural environments. Additionally, we identify that existing anatomical keypoint detectors are insufficient for accurate pose retargeting onto arbitrary avatars. To overcome this, we present a lookup table based on a deep pose estimation method using a synthetic collection of diverse actions rigged avatars perform. Our experiments demonstrate the effectiveness and efficiency of this lookup table-based retargeting approach. Overall, we propose a comprehensive framework with systematically synthesized datasets for lifting poses from 2D to 3D and then utilize this to re-target motion from wild settings onto arbitrary avatars.

Figures (3)

• Figure 1: Process flow of the proposed methodology. For a given natural image, we first use pre-trained 2D pose estimation techniques to obtain 2D keypoints in the image. Our attention-based simple MLP architecture, trained on a synthetic dataset, effectively lifts these normalised 2D keypoints into a partial soft 3D pose, as illustrated in (a). We then match this partial 3D pose to the closest deep pose from a look-up table, which includes a diverse set of 3D poses derived from synthetic motion sequences. The Deep 3D Pose provides the necessary information to transfer the pose from the image onto an avatar model, as demonstrated in (b). Zooming in is recommended for better clarity.
• Figure 2: Lifting and retargeting of 3D Pose on rigged models obtained by the proposed framework. We present results on two assets: Macaque and Horse.
• Figure 3: (a) We illustrate our data acquisition setup in Unity, where the subject is positioned at the origin, and the black markers trace the camera's trajectory during data collection. The camera consistently faces the subject, with the field of view calibrated to keep the subject fully visible at all times. (b) and (c) display $k=10$ clusters of 3D poses derived from the look-up table generated from synthetic Macaque and Horse models. In each case, over 60% of the instances in each cluster correspond to a specific action set. A sample image from each cluster's most frequent action sequence is shown. The scatter points represent the values of the two most dominant principal components obtained from PCA on the look-up table, with colors indicating the cluster to which each scatter point belongs. The corresponding color bar shows the most prominent action in the cluster. Zooming in is recommended for better clarity.