RUMPL: Ray-Based Transformers for Universal Multi-View 2D to 3D Human Pose Lifting

TL;DR

RUMPL introduces a universal, camera-agnostic approach to multi-view 3D human pose lifting by representing 2D keypoints as 3D rays and fusing information across views with a ray-aware transformer. Trained solely on synthetic 2D-3D pairs generated from AMASS meshes, the method eliminates dependence on specific camera configurations and ground-truth multi-view data, enabling deployment in arbitrary environments. Extensive experiments across Human3.6M, CMU Panoptic, and RICH demonstrate large MPJPE improvements over triangulation and strong baselines, with ablations confirming the value of the 3D ray representation and synthetic-data training. The work provides practical gains in robustness, speed, and scalability for real-world, in-the-wild 3D pose estimation.

Abstract

Estimating 3D human poses from 2D images remains challenging due to occlusions and projective ambiguity. Multi-view learning-based approaches mitigate these issues but often fail to generalize to real-world scenarios, as large-scale multi-view datasets with 3D ground truth are scarce and captured under constrained conditions. To overcome this limitation, recent methods rely on 2D pose estimation combined with 2D-to-3D pose lifting trained on synthetic data. Building on our previous MPL framework, we propose RUMPL, a transformer-based 3D pose lifter that introduces a 3D ray-based representation of 2D keypoints. This formulation makes the model independent of camera calibration and the number of views, enabling universal deployment across arbitrary multi-view configurations without retraining or fine-tuning. A new View Fusion Transformer leverages learned fused-ray tokens to aggregate information along rays, further improving multi-view consistency. Extensive experiments demonstrate that RUMPL reduces MPJPE by up to 53% compared to triangulation and over 60% compared to transformer-based image-representation baselines. Results on new benchmarks, including in-the-wild multi-view and multi-person datasets, confirm its robustness and scalability. The framework's source code is available at https://github.com/aghasemzadeh/OpenRUMPL

Figures (4)

• Figure 1: RUMPL takes 3D rays of all keypoints from all views, fuses the rays associated to a given keypoint and then jointly considers the whole set of keypoints using a transformer network.
• Figure 2: Our Mesh-based Human Pose Dataset Generator (MHP) takes 3D mesh vertices and the limits of camera and person's displacement in a given room as inputs. It randomly positions the mesh and $N$ cameras in the scene and returns the 3D pose ground truth and one 2D pose per camera view. Each 2D pose is estimated by an off-the-shelf 2D pose estimation model, applied to the image of the mesh rendered in the corresponding view. The 3D keypoint regressor defines the 3D pose ground truth in a way that is consistent with the definition of ground truth at testing. This makes the 2D-3D pairs of human pose appropriate for training an accurate and robust RUMPL, able to turn the 2D poses computed by the off-the-shelf pose estimator into a 3D skeleton that is consistent with the 3D keypoints expected at inference. Note that the triangles represent the location of the cameras, the light blue cube illustrates the camera limits, and the blue surface depicts the person's limits in the room.
• Figure 3: Qualitative results. In the top row, the GT and MMPOSE 2D represent the ground truth taken from the CMU dataset and the predicted 2D pose from the off-the-shelf 2D pose estimator, respectively. In the middle row, GT shows the 3D ground truth from the CMU dataset, and the other pose corresponds to the output of triangulation when the top row is given as inputs. In the bottom row, the outputs of MPL coupled with random camera setup (RCS) and ray representation and RUMPL are shown.
• Figure 4: The change in MPJPE when changing the composition of the cameras. The angles are calculated in a 2-view camera setup. The cameras are placed on a sphere with radius of 6 $m$ and center of $[0,0,0]$. The heights are 2.2, 3.2, and 4 $m$.