Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics

Sandika Biswas, Kian Izadpanah, Hamid Rezatofighi

TL;DR

This work addresses the need for realistic, multi-person 3D pose datasets for robotics by introducing JRDB-Pose3D. The dataset provides temporally consistent SMPL-based 3D pose and shape annotations with per-track IDs, built on the JRDB robot-navigation data, and inherits extensive JRDB context like 2D poses, social/group annotations, and full-scene segmentation. A five-stage annotation pipeline combines CameraHMR initialization, global pose localization, shape consistency, local pose refinement, and manual verification to ensure accuracy under heavy occlusions and out-of-frame cases. By offering dense, real-world, robot-view data, JRDB-Pose3D enables robust evaluation of multi-person pose estimation, tracking, and pose-aware activity recognition in crowded environments, with broad applicability to autonomous navigation and human-robot interaction.

Abstract

Real-world scenes are inherently crowded. Hence, estimating 3D poses of all nearby humans, tracking their movements over time, and understanding their activities within social and environmental contexts are essential for many applications, such as autonomous driving, robot perception, robot navigation, and human-robot interaction. However, most existing 3D human pose estimation datasets primarily focus on single-person scenes or are collected in controlled laboratory environments, which restricts their relevance to real-world applications. To bridge this gap, we introduce JRDB-Pose3D, which captures multi-human indoor and outdoor environments from a mobile robotic platform. JRDB-Pose3D provides rich 3D human pose annotations for such complex and dynamic scenes, including SMPL-based pose annotations with consistent body-shape parameters and track IDs for each individual over time. JRDB-Pose3D contains, on average, 5-10 human poses per frame, with some scenes featuring up to 35 individuals simultaneously. The proposed dataset presents unique challenges, including frequent occlusions, truncated bodies, and out-of-frame body parts, which closely reflect real-world environments. Moreover, JRDB-Pose3D inherits all available annotations from the JRDB dataset, such as 2D pose, information about social grouping, activities, and interactions, full-scene semantic masks with consistent human- and object-level tracking, and detailed annotations for each individual, such as age, gender, and race, making it a holistic dataset for a wide range of downstream perception and human-centric understanding tasks.

JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics

TL;DR

This work addresses the need for realistic, multi-person 3D pose datasets for robotics by introducing JRDB-Pose3D. The dataset provides temporally consistent SMPL-based 3D pose and shape annotations with per-track IDs, built on the JRDB robot-navigation data, and inherits extensive JRDB context like 2D poses, social/group annotations, and full-scene segmentation. A five-stage annotation pipeline combines CameraHMR initialization, global pose localization, shape consistency, local pose refinement, and manual verification to ensure accuracy under heavy occlusions and out-of-frame cases. By offering dense, real-world, robot-view data, JRDB-Pose3D enables robust evaluation of multi-person pose estimation, tracking, and pose-aware activity recognition in crowded environments, with broad applicability to autonomous navigation and human-robot interaction.

Abstract

Real-world scenes are inherently crowded. Hence, estimating 3D poses of all nearby humans, tracking their movements over time, and understanding their activities within social and environmental contexts are essential for many applications, such as autonomous driving, robot perception, robot navigation, and human-robot interaction. However, most existing 3D human pose estimation datasets primarily focus on single-person scenes or are collected in controlled laboratory environments, which restricts their relevance to real-world applications. To bridge this gap, we introduce JRDB-Pose3D, which captures multi-human indoor and outdoor environments from a mobile robotic platform. JRDB-Pose3D provides rich 3D human pose annotations for such complex and dynamic scenes, including SMPL-based pose annotations with consistent body-shape parameters and track IDs for each individual over time. JRDB-Pose3D contains, on average, 5-10 human poses per frame, with some scenes featuring up to 35 individuals simultaneously. The proposed dataset presents unique challenges, including frequent occlusions, truncated bodies, and out-of-frame body parts, which closely reflect real-world environments. Moreover, JRDB-Pose3D inherits all available annotations from the JRDB dataset, such as 2D pose, information about social grouping, activities, and interactions, full-scene semantic masks with consistent human- and object-level tracking, and detailed annotations for each individual, such as age, gender, and race, making it a holistic dataset for a wide range of downstream perception and human-centric understanding tasks.
Paper Structure (11 sections, 5 equations, 4 figures, 1 table)

This paper contains 11 sections, 5 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Datasets
  3. Single human pose estimation
  4. Multiple Human Pose Estimation
  5. 3D Skeleton-based Annotations
  6. 3D Human Pose and Shape (HPS) Annotations
  7. JRDB-Pose3D Dataset
  8. Dataset overview
  9. Annotation
  10. Uniqueness of JRDB-Pose3D
  11. Conclusion

Figures (4)

  • Figure 1: Example visualization of the proposed JRDB-pose3D dataset with an indoor - 'tressider-2019-04-26_1' (first row) and outdoor 'food-trucks-2019-02-12_0' (second row) multi-person scene.
  • Figure 2: Statistics of the per-frame number of poses for (a) Ours and (b) WorldPose dataset.
  • Figure 3: Kernel Density Estimates (KDEs) of the spatial distribution of people around the robot, i.e., distance from the camera and angular positions, across (a) Ours and (b) WorldPose dataset.
  • Figure 4: (a) Statistics of occlusion and out-of-frame poses in the JRDB-Pose3D dataset in terms of the number of occluded, invisible, and visible 2D joints. (b) Statistics of poses in the JRDB-Pose3D dataset according to the difficulty of 3D pose recovery, classified as easy, medium, and hard.