Unsupervised 2D-3D lifting of non-rigid objects using local constraints

TL;DR

The paper tackles unsupervised 2D-to-3D lifting of non-rigid objects by introducing ALLRAP, which enforces locally low-rank constraints through a matrix inpainting framework. It replaces global priors with neighborhood-based constraints and two unsupervised losses—the subset loss and the occlusion loss—implemented via a parameter-efficient MLP-Mixer network. The approach achieves state-of-the-art performance on the S-Up3D dataset with over 70% reconstruction-error reduction and demonstrates strong one-shot lifting on DeformingThings4D and ZJU-Mocap sequences. This locally constrained, unsupervised framework enables robust 3D reconstruction from partially-occluded 2D keypoints across diverse datasets and camera settings, with an open-source release planned.

Abstract

For non-rigid objects, predicting the 3D shape from 2D keypoint observations is ill-posed due to occlusions, and the need to disentangle changes in viewpoint and changes in shape. This challenge has often been addressed by embedding low-rank constraints into specialized models. These models can be hard to train, as they depend on finding a canonical way of aligning observations, before they can learn detailed geometry. These constraints have limited the reconstruction quality. We show that generic, high capacity models, trained with an unsupervised loss, allow for more accurate predicted shapes. In particular, applying low-rank constraints to localized subsets of the full shape allows the high capacity to be suitably constrained. We reduce the state-of-the-art reconstruction error on the S-Up3D dataset by over 70%.

Figures (5)

• Figure 1: Unsupervised losses Starting from a randomly-initialized generic deep network, we iteratively learn from batches of partially-occluded 2D-annotated training samples. Our training use two unsupervised, batch-wise losses. The subset loss, see Section \ref{['sec:tuple_loss']}, acts on the batch of noisy 3D reconstructions. It selects a subset of nearby keypoints, aligns the sub-shapes by rotation and translation, and finally measures the size of the non-rigid motion using the log-product of the singular value decomposition of the residual error matrix, i.e. the log Gramian determinant. This encourages the model to predict body parts that are as consistent as possible. The occlusion loss, see Section \ref{['sec:occlusion']}, encourages a weak negative correlation between the binary keypoint-visibility annotations and the predicted depths. This uses the fact that visible keypoints often hide other keypoints because they are closer to the camera.
• Figure 2: Subset loss training-time efficiency Plotting the results from Table \ref{['tbl:tuple_selection']}, we see that selecting neigborhoods of 32 keypoints works well, whilst being reasonably fast at training time.
• Figure 3: ALLRAP S-Up3D test-set reconstructions. (a) We break down the MPJPE error into errors in the camera plane due to occlusion, and errors in the predicted depths. (b) shows a test case with median errors in both components. (c) show the test cases with maximum errors in the camera plane; a leg is occluded and in an unusually high position. (d) shows the worst depth error: it is an unusual case as the body is observed from an over-the-head position. Errors are show in using, blue, green and red lines, respectively.
• Figure 4: ALLRAP DeformingThings4D one-shot reconstruction Results from training ALLRAP on a sequence of 34 frames from the DeformingThings4D dataset. The reconstructions are shown from a top-down view, with errors shown with red lines.
• Figure 5: ALLRAP ZJU-MoCap one-shot reconstructions Results from training ALLRAP on a single sequence of 570 frames from the ZJU-MoCap dataset annotated using FastCapture. Errors compared to multicamera reconstruction are shown using red lines.