Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape

Jiacong Xu, Yi Zhang, Jiawei Peng, Wufei Ma, Artur Jesslen, Pengliang Ji, Qixin Hu, Jiehua Zhang, Qihao Liu, Jiahao Wang, Wei Ji, Chen Wang, Xiaoding Yuan, Prakhar Kaushik, Guofeng Zhang, Jie Liu, Yushan Xie, Yawen Cui, Alan Yuille, Adam Kortylewski

TL;DR

Animal3D addresses the lack of diverse, high-quality 3D animal pose data by providing the first mammal-focused benchmark with $3379$ images across $40$ species, $26$ keypoints, and SMAL $(eta, \alpha, t)$ parameters. A semi-interactive annotation pipeline and a robust SMAL fitting procedure yield accurate 3D pose/shape annotations from 2D keypoints and silhouettes, enabling multi-task evaluation. Through studies of supervised, synthetic-to-real, and human-pretrained baselines, the work reveals persistent cross-species generalization gaps and demonstrates the value of synthetic pre-training in improving 3D animal pose estimation. The Animal3D resource, released publicly, aims to accelerate research in animal behavior understanding and conservation by facilitating robust, cross-species 3D pose and shape estimation.

Abstract

Accurately estimating the 3D pose and shape is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. However, research in this area is held back by the lack of a comprehensive and diverse dataset with high-quality 3D pose and shape annotations. In this paper, we propose Animal3D, the first comprehensive dataset for mammal animal 3D pose and shape estimation. Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model. All annotations were labeled and checked manually in a multi-stage process to ensure highest quality results. Based on the Animal3D dataset, we benchmark representative shape and pose estimation models at: (1) supervised learning from only the Animal3D data, (2) synthetic to real transfer from synthetically generated images, and (3) fine-tuning human pose and shape estimation models. Our experimental results demonstrate that predicting the 3D shape and pose of animals across species remains a very challenging task, despite significant advances in human pose estimation. Our results further demonstrate that synthetic pre-training is a viable strategy to boost the model performance. Overall, Animal3D opens new directions for facilitating future research in animal 3D pose and shape estimation, and is publicly available.

Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape

TL;DR

Animal3D addresses the lack of diverse, high-quality 3D animal pose data by providing the first mammal-focused benchmark with images across species, keypoints, and SMAL parameters. A semi-interactive annotation pipeline and a robust SMAL fitting procedure yield accurate 3D pose/shape annotations from 2D keypoints and silhouettes, enabling multi-task evaluation. Through studies of supervised, synthetic-to-real, and human-pretrained baselines, the work reveals persistent cross-species generalization gaps and demonstrates the value of synthetic pre-training in improving 3D animal pose estimation. The Animal3D resource, released publicly, aims to accelerate research in animal behavior understanding and conservation by facilitating robust, cross-species 3D pose and shape estimation.

Abstract

Accurately estimating the 3D pose and shape is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. However, research in this area is held back by the lack of a comprehensive and diverse dataset with high-quality 3D pose and shape annotations. In this paper, we propose Animal3D, the first comprehensive dataset for mammal animal 3D pose and shape estimation. Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model. All annotations were labeled and checked manually in a multi-stage process to ensure highest quality results. Based on the Animal3D dataset, we benchmark representative shape and pose estimation models at: (1) supervised learning from only the Animal3D data, (2) synthetic to real transfer from synthetically generated images, and (3) fine-tuning human pose and shape estimation models. Our experimental results demonstrate that predicting the 3D shape and pose of animals across species remains a very challenging task, despite significant advances in human pose estimation. Our results further demonstrate that synthetic pre-training is a viable strategy to boost the model performance. Overall, Animal3D opens new directions for facilitating future research in animal 3D pose and shape estimation, and is publicly available.
Paper Structure (16 sections, 3 equations, 5 figures, 2 tables)

This paper contains 16 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Animal Pose Estimation
  4. Human Pose Estimation
  5. Animal3D Dataset
  6. Data Collection
  7. Data Annotation
  8. Fitting Animals to Images
  9. Data Summary
  10. Experiments
  11. Supervised Animal Shape and Pose Regression
  12. From Synthetic to Real
  13. From Human to Animal
  14. Qualitative Results
  15. Discussion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Samples from the proposed Animal3D dataset. Our dataset contains a diverse range of animal species with high-quality annotations of shape and pose parameters using the popular SMAL zuffi20173d model.
  • Figure 2: Data annotation pipeline for Animal3D. The process consists of three stages: Image Filtering, Semi-Interactive Annotation, and Data Integration. The data is sourced and filtered to obtain an initial set of images. During the Semi-Interactive Annotation, annotators submitted their annotation to the server to fit the SMAL model and render the results on the images. Then a set of inspectors examined the fitting results and send the bad-fitting images back to the annotator for revision. This process is repeated multiple times. Images that constantly lead to bad-fitting results are removed.
  • Figure 3: Visualization of the 26 keypoints that are annotated in the Animal3D model. Other popular 2D datasets only annotate the visible keypoints, while we ask the annotators to guess the location of occluded or truncated body parts based on their annotation experience, which significantly improves the fitting performance of SMAL model.
  • Figure 4: Example images from our synthetic dataset that is used for pre-training the animal pose estimation baselines. We simulate all species from the Animal3D dataset using the SMALR model in varying poses, shapes, and background images.
  • Figure 5: Visualization of regression performance of human and animal pose estimation models. The columns from left to right refer to the input image, the groundtruth from Animal3D, regression results for HMR, HMR pretrained by synthetic and human data, PARE and WLDO pretrained by synthetic data, respectively.