Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UnrealPose: Leveraging Game Engine Kinematics for Large-Scale Synthetic Human Pose Data

Joshua Kawaguchi, Saad Manzur, Emily Gao Wang, Maitreyi Sinha, Bryan Vela, Yunxi Wang, Brandon Vela, Wayne B. Hayes

TL;DR

UnrealPose tackles the lack of large-scale, accurately labeled 3D human pose data by introducing UnrealPose-Gen, a UE5-based pipeline that renders engine-native skeletal joints with full camera calibration and COCO-style annotations. The system enables offline rendering via Movie Render Queue and online rendering within gameplay, and it outputs per-frame 3D joints (world and camera coordinates), 2D keypoints with visibility, bounding boxes, and segmentation masks. UnrealPose-1M comprises approximately one million annotated frames across coherent and randomized sequences, designed to maximize viewpoint diversity and interaction-rich motions. Across four tasks—2D keypoint detection, 2D→3D lifting, image→3D pose regression, and instance segmentation—the synthetic data show strong geometric fidelity and cross-domain consistency, supporting domain adaptation and synthetic-to-real transfer, with the dataset and generator publicly released for broader use.

Abstract

Diverse, accurately labeled 3D human pose data is expensive and studio-bound, while in-the-wild datasets lack known ground truth. We introduce UnrealPose-Gen, an Unreal Engine 5 pipeline built on Movie Render Queue for high-quality offline rendering. Our generated frames include: (i) 3D joints in world and camera coordinates, (ii) 2D projections and COCO-style keypoints with occlusion and joint-visibility flags, (iii) person bounding boxes, and (iv) camera intrinsics and extrinsics. We use UnrealPose-Gen to present UnrealPose-1M, an approximately one million frame corpus comprising eight sequences: five scripted "coherent" sequences spanning five scenes, approximately 40 actions, and five subjects; and three randomized sequences across three scenes, approximately 100 actions, and five subjects, all captured from diverse camera trajectories for broad viewpoint coverage. As a fidelity check, we report real-to-synthetic results on four tasks: image-to-3D pose, 2D keypoint detection, 2D-to-3D lifting, and person detection/segmentation. Though time and resources constrain us from an unlimited dataset, we release the UnrealPose-1M dataset, as well as the UnrealPose-Gen pipeline to support third-party generation of human pose data.

UnrealPose: Leveraging Game Engine Kinematics for Large-Scale Synthetic Human Pose Data

TL;DR

UnrealPose tackles the lack of large-scale, accurately labeled 3D human pose data by introducing UnrealPose-Gen, a UE5-based pipeline that renders engine-native skeletal joints with full camera calibration and COCO-style annotations. The system enables offline rendering via Movie Render Queue and online rendering within gameplay, and it outputs per-frame 3D joints (world and camera coordinates), 2D keypoints with visibility, bounding boxes, and segmentation masks. UnrealPose-1M comprises approximately one million annotated frames across coherent and randomized sequences, designed to maximize viewpoint diversity and interaction-rich motions. Across four tasks—2D keypoint detection, 2D→3D lifting, image→3D pose regression, and instance segmentation—the synthetic data show strong geometric fidelity and cross-domain consistency, supporting domain adaptation and synthetic-to-real transfer, with the dataset and generator publicly released for broader use.

Abstract

Diverse, accurately labeled 3D human pose data is expensive and studio-bound, while in-the-wild datasets lack known ground truth. We introduce UnrealPose-Gen, an Unreal Engine 5 pipeline built on Movie Render Queue for high-quality offline rendering. Our generated frames include: (i) 3D joints in world and camera coordinates, (ii) 2D projections and COCO-style keypoints with occlusion and joint-visibility flags, (iii) person bounding boxes, and (iv) camera intrinsics and extrinsics. We use UnrealPose-Gen to present UnrealPose-1M, an approximately one million frame corpus comprising eight sequences: five scripted "coherent" sequences spanning five scenes, approximately 40 actions, and five subjects; and three randomized sequences across three scenes, approximately 100 actions, and five subjects, all captured from diverse camera trajectories for broad viewpoint coverage. As a fidelity check, we report real-to-synthetic results on four tasks: image-to-3D pose, 2D keypoint detection, 2D-to-3D lifting, and person detection/segmentation. Though time and resources constrain us from an unlimited dataset, we release the UnrealPose-1M dataset, as well as the UnrealPose-Gen pipeline to support third-party generation of human pose data.
Paper Structure (18 sections, 7 figures, 1 table)

This paper contains 18 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human Pose Datasets
  4. Learning 3D from 2D
  5. Synthetic Human Pose Datasets
  6. Summary
  7. Methods
  8. UnrealPose-Gen
  9. UnrealPose-1M
  10. Experiments
  11. Image → 2D Keypoint Detection
  12. Top-down.
  13. Bottom-up.
  14. 2D → 3D Lifting
  15. Image → 3D Joint Regression
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Visualization of annotations in UnrealPose-1M. The first two panels show the full-resolution RGB frame and its instance mask overlay. The remaining panels show cropped regions for easier viewing, containing person bounding boxes, standard 2D keypoints, and COCO keypoints with occlusion flags (green).
  • Figure 2: Example camera configuration. We vary FOV (30°--90°), camera height (ground to overhead), and distance to subject, producing unconventional viewpoints including ground-level and steep overhead angles rarely present in existing datasets.
  • Figure 3: Per-joint MPJPE distribution across 16 body joints on evaluation of PoseAug.
  • Figure 4: Single-person synthetic image (left) and the corresponding 3D pose predicted by MeTRAbs (right).
  • Figure 5: Two-person synthetic image (left) and the corresponding 3D poses predicted by MeTRAbs (right).
  • ...and 2 more figures