Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud

TL;DR

The paper tackles multi-person 3D pose estimation from multi-view video under strong occlusions by proposing the Joint Cloud, a comprehensive set of 3D joint candidates derived from all same-type 2D detections across views. A three-encoder Transformer pipeline, JCSAT, with Optimal Token Attention Path (OTAP) selectively aggregates trajectory, skeletal, and angular cues to regress robust 3D motion; a joint-type conversion module and joint masking further improve robustness. The authors introduce BUMocap-X and demonstrate state-of-the-art performance across Shelf, Campus, BUMocap, and BUMocap-X datasets, particularly under severe occlusions, along with extensive ablations. The work advances practical multi-view motion capture by maximizing utilization of 2D detections and providing a scalable, occlusion-resilient framework with strong generalization potential. Overall, this method enables more reliable, camera-robust 3D human pose capture in crowded, occluded scenarios and offers a valuable dataset for evaluation.

Abstract

Multi-person motion capture over sparse angular observations is a challenging problem under interference from both self- and mutual-occlusions. Existing works produce accurate 2D joint detection, however, when these are triangulated and lifted into 3D, available solutions all struggle in selecting the most accurate candidates and associating them to the correct joint type and target identity. As such, in order to fully utilize all accurate 2D joint location information, we propose to independently triangulate between all same-typed 2D joints from all camera views regardless of their target ID, forming the Joint Cloud. Joint Cloud consist of both valid joints lifted from the same joint type and target ID, as well as falsely constructed ones that are from different 2D sources. These redundant and inaccurate candidates are processed over the proposed Joint Cloud Selection and Aggregation Transformer (JCSAT) involving three cascaded encoders which deeply explore the trajectile, skeletal structural, and view-dependent correlations among all 3D point candidates in the cross-embedding space. An Optimal Token Attention Path (OTAP) module is proposed which subsequently selects and aggregates informative features from these redundant observations for the final prediction of human motion. To demonstrate the effectiveness of JCSAT, we build and publish a new multi-person motion capture dataset BUMocap-X with complex interactions and severe occlusions. Comprehensive experiments over the newly presented as well as benchmark datasets validate the effectiveness of the proposed framework, which outperforms all existing state-of-the-art methods, especially under challenging occlusion scenarios.

Figures (16)

• Figure 1: Demonstration of 2D detections from state-of-the-art 2D pose detectors cao2019openposefang2022alphaposesun2019deep at the same timestamp on Shelf dataset belagiannis2014multiple. Although both produce the most accurate results, there are still some defects notated by red dotted boxes regarding the limbs: 1) assigned to incorrect target identity, 2) assigned to incorrect joint type.
• Figure 1: Demonstration of the triangulation procedure in the Joint Cloud construction pipeline. Three views are presented in the scene. The 2D detections of the "Left Knee" are depicted by yellow dots. From one of the view pair combinations, the 3D triangulated candidates are shown in orange dots. Orange dashed lines symbolize the ray cast from a camera centre to the 2D point on the camera plane. For clarity, only four candidates are illustrated while there should be twelve candidates in total.
• Figure 2: Qualitative comparison with the most recent multi-view multi-person motion capture frameworks on our newly collected dataset BUMocap-X. We demonstrate the projection and rendered results as well as the manual-labelled ground truth (from left to right: 4DAssocGraph zhang20204d, DMAE jiang2022dual, TEMPO choudhury2023tempo, ours, the Joint Cloud, and the ground truth). We provide the rendered Joint Cloud of Actor-3 in ten frames. Same-typed joints are connected to indicate the trajectory. The proposed model reasons on trajectile, structural skeletal and angular information from the Joint Cloud and produces better predictions than existing frameworks, especially under occlusion scenarios.
• Figure 2: Demonstration of the mid-hip clustering procedure in the Joint Cloud construction pipeline. Yellow dots represent the 2D detections of "mid-hip". Orange dots represent the 3D triangulated candidates. Red dots represent the centroid of each cluster. We utilize the centroid as each individual's body centre indicating the corresponding Joint Cloud subset's centre.
• Figure 3: The proposed JCSAT pipeline for efficiently extracting kinematic skeletal structures from multi-view Joint Clouds. For every multi-view camera pair (e.g., $\alpha\gamma$), the Trajectory Encoder and Structure Encoder aggregate trajectile and skeletal features from the triangulated Joint Cloud in parallel. The T-OTAP and S-OTAP modules select the optimal tokens along the joint candidate axis (joint candidates are constituents of the Joint Cloud, with each joint having multiple candidates triangulated from different camera view pairs) in a differentiable manner. By traversing through all joints and frames, these candidate feature tokens are then blended element-wise into cross-embedding tokens. The cross-embedding tokens are then gathered across the axis of camera view pairs. A View Encoder extracts angular features based on which the subsequent V-OTAP module selects the most representative token for every joint across all frames. Ultimately, an MLP head module decodes the refined trajectory. The three OTAP modules enable comprehensive and efficient optimization of the entire system from end to end.