Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset

Claire McLean, Makenzie Meendering, Tristan Swartz, Orri Gabbay, Alexandra Olsen, Rachel Jacobs, Nicholas Rosen, Philippe de Bree, Tony Garcia, Gadsden Merrill, Jake Sandakly, Julia Buffalini, Neham Jain, Steven Krenn, Moneish Kumar, Dejan Markovic, Evonne Ng, Fabian Prada, Andrew Saba, Siwei Zhang, Vasu Agrawal, Tim Godisart, Alexander Richard, Michael Zollhoefer

TL;DR

Embody 3D introduces a large-scale multimodal 3D motion and behavior dataset that unifies extensive motion capture with audio and text annotations across single- and multi-person scenarios. It describes a comprehensive data collection system with 80 cameras and a 640-channel microphone array, along with an end-to-end processing pipeline for calibration, shape estimation (SMPL-X), multi-person 3D pose, and beamformed speech per participant. The work emphasizes overcoming existing trade-offs in motion datasets by achieving scale while preserving tracking quality and modality richness, enabling robust motion understanding, behavior modeling, and virtual-human research. The dataset covers Charades, Hand Interactions, Locomotion, Dyadic and Multi-Person Conversations, Scenarios, and Day-in-the-Life scenarios in an apartment-like space, totaling 500 hours and 54M frames from 439 participants.

Abstract

The Codec Avatars Lab at Meta introduces Embody 3D, a multimodal dataset of 500 individual hours of 3D motion data from 439 participants collected in a multi-camera collection stage, amounting to over 54 million frames of tracked 3D motion. The dataset features a wide range of single-person motion data, including prompted motions, hand gestures, and locomotion; as well as multi-person behavioral and conversational data like discussions, conversations in different emotional states, collaborative activities, and co-living scenarios in an apartment-like space. We provide tracked human motion including hand tracking and body shape, text annotations, and a separate audio track for each participant.

Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset

TL;DR

Embody 3D introduces a large-scale multimodal 3D motion and behavior dataset that unifies extensive motion capture with audio and text annotations across single- and multi-person scenarios. It describes a comprehensive data collection system with 80 cameras and a 640-channel microphone array, along with an end-to-end processing pipeline for calibration, shape estimation (SMPL-X), multi-person 3D pose, and beamformed speech per participant. The work emphasizes overcoming existing trade-offs in motion datasets by achieving scale while preserving tracking quality and modality richness, enabling robust motion understanding, behavior modeling, and virtual-human research. The dataset covers Charades, Hand Interactions, Locomotion, Dyadic and Multi-Person Conversations, Scenarios, and Day-in-the-Life scenarios in an apartment-like space, totaling 500 hours and 54M frames from 439 participants.

Abstract

The Codec Avatars Lab at Meta introduces Embody 3D, a multimodal dataset of 500 individual hours of 3D motion data from 439 participants collected in a multi-camera collection stage, amounting to over 54 million frames of tracked 3D motion. The dataset features a wide range of single-person motion data, including prompted motions, hand gestures, and locomotion; as well as multi-person behavioral and conversational data like discussions, conversations in different emotional states, collaborative activities, and co-living scenarios in an apartment-like space. We provide tracked human motion including hand tracking and body shape, text annotations, and a separate audio track for each participant.
Paper Structure (9 sections, 4 figures, 2 tables)

This paper contains 9 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Dataset Details
  3. Collection System and Dataset Acquisition
  4. Data Processing
  5. Synchronization and Calibration
  6. Participant Shape Estimation
  7. Multi-Person Pose Estimation
  8. Beamforming
  9. Quality Assurance

Figures (4)

  • Figure 1: Example frames from each dataset subcategory. For single-person categories like Charades, Hand Interactions, and Locomotion, we collect multiple participants at the same time for better throughput in terms of individual participant hours. For interactive subcategories like conversations or scenarios, on the contrary, all participants in the scene interact with each other.
  • Figure 2: End to end processing pipeline for generating multi-person poses and audio from raw collected data.
  • Figure 3: Each participant performs calibration poses to obtain person specific body shape and reference face images.
  • Figure 4: We generate 2d keypoint detections within the bounding boxes of a multi-person detector (left). Keypoints are then matched to participants using geometric and appearance cues (right).