Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions

TL;DR

A novel markerless algorithm is proposed to track 3D human poses in severe occlusion and close interaction to obtain annotations for human detection, tracking, 2D/3D pose estimation, and mesh recovery for closely interacting subjects in Harmony4D.

Abstract

Understanding how humans interact with each other is key to building realistic multi-human virtual reality systems. This area remains relatively unexplored due to the lack of large-scale datasets. Recent datasets focusing on this issue mainly consist of activities captured entirely in controlled indoor environments with choreographed actions, significantly affecting their diversity. To address this, we introduce Harmony4D, a multi-view video dataset for human-human interaction featuring in-the-wild activities such as wrestling, dancing, MMA, and more. We use a flexible multi-view capture system to record these dynamic activities and provide annotations for human detection, tracking, 2D/3D pose estimation, and mesh recovery for closely interacting subjects. We propose a novel markerless algorithm to track 3D human poses in severe occlusion and close interaction to obtain our annotations with minimal manual intervention. Harmony4D consists of 1.66 million images and 3.32 million human instances from more than 20 synchronized cameras with 208 video sequences spanning diverse environments and 24 unique subjects. We rigorously evaluate existing state-of-the-art methods for mesh recovery and highlight their significant limitations in modeling close interaction scenarios. Additionally, we fine-tune a pre-trained HMR2.0 model on Harmony4D and demonstrate an improved performance of 54.8% PVE in scenes with severe occlusion and contact. Code and data are available at https://jyuntins.github.io/harmony4d/.

Figures (10)

• Figure 1: Overview of Harmony4D setup. (a) Multiple synchronized and calibrated cameras capture the contact interaction in wrestling. (b) We align all cameras into a gravity-aligned metric world coordinate system. (c) Our processing obtains per-view instance segmentation masks along with 3D keypoints. (d) Reconstructed ground-truth meshes after multi-step collision optimization.
• Figure 2: Dataset Composition. Harmony4D consists of diverse, dynamic activities such as dancing, karate, MMA, and wrestling, all captured in in-the-wild settings.
• Figure 3: Data Distribution. The dynamic activities in the Harmony4D dataset cover all area for the SMPL body model. We visualize the most frequent body parts in contact during interactions as a normalized heatmap.
• Figure 4: Overview of Harmony4D processing setup. Given a multi-view RGB video sequence, we divide it into pre-contact and post-contact stages. We estimate per-subject 3D poses in the pre-contact stage khirodkar2023ego as initialization. The post-contact stage uses sequential processing involving 3D pose forecasting with a human motion model, per-view 2D point-conditioned instance segmentation, and mask-conditioned 2D pose estimation, followed by multi-view triangulation and mesh fitting.
• Figure 5: (Left) Point conditioned instance segmentation. Projected 3D keypoints as positive or negative prompts. (Right) Comparison of ViTPose xu2022vitpose with mask-conditioned 2D pose estimation.