On the Role of Rotation Equivariance in Monocular 3D Human Pose Estimation

TL;DR

This work addresses monocular 3D human pose estimation by focusing on the 2D to 3D lifting step and the role of rotation equivariance. It argues that rotation equivariance can be effectively learned via augmentation rather than enforced by design, and studies three model categories—fully equivariant, hybrid, and vanilla—across multiple datasets using MPJPE and PA-MPJPE. The key finding is that vanilla models trained with rotation augmentation (vanilla+aug) generally outperform strictly equivariant or hybrid designs, with strong generalization and faster training/inference. The results motivate using augmentation-based rotation awareness in practical 2D-to-3D lifting for human pose estimation and point to broader implications for symmetry-aware learning in ill-posed vision problems.

Abstract

Estimating 3D from 2D is one of the central tasks in computer vision. In this work, we consider the monocular setting, i.e. single-view input, for 3D human pose estimation (HPE). Here, the task is to predict a 3D point set of human skeletal joints from a single 2D input image. While by definition this is an ill-posed problem, recent work has presented methods that solve it with up to several-centimetre error. Typically, these methods employ a two-step approach, where the first step is to detect the 2D skeletal joints in the input image, followed by the step of 2D-to-3D lifting. We find that common lifting models fail when encountering a rotated input. We argue that learning a single human pose along with its in-plane rotations is considerably easier and more geometrically grounded than directly learning a point-to-point mapping. Furthermore, our intuition is that endowing the model with the notion of rotation equivariance without explicitly constraining its parameter space should lead to a more straightforward learning process than one with equivariance by design. Utilising the common HPE benchmarks, we confirm that the 2D rotation equivariance per se improves the model performance on human poses akin to rotations in the image plane, and can be efficiently and straightforwardly learned by augmentation, outperforming state-of-the-art equivariant-by-design methods.

Figures (8)

• Figure 1: Teaser: Learning rotation equivariance improves the performance of a 2D$\rightarrow$3D lifting model in monocular human pose estimation for motions akin to rotations in the image plane. In contrast, typical lifting models lack this notion of equivariance. Furthermore, models that are equivariant by design over-constrain the learning process; both result in poor 3D pose predictions. Blue, green, and orange, respectively, encode the right side, middle, and left side of the human body.
• Figure 2: Geometric consistency in the lifting model: rotations in the image plane correspond to the rotations about the optical axis in the camera coordinate system.
• Figure 3: Hybrid model architecture outline: the $xy$-outputs are rotation-equivariant, whereas the $z$-coordinate prediction is non-equivariant. In the default setting (no dotted line), the input is fed into both models in parallel; for the ablation study (with the dotted line), the input to the non-equivariant baseline is the output of the first equivariant layer of the 2D equivariant model.
• Figure 4: Samples from the datasets used in our experiments. SportsCap contains poses with realistic full-body rotations that do not exist in the Human3.6M and the MPII-INF-3DHP datasets.
• Figure 5: Representative example of the performance of different models on a SportsCap sample: Learned equivariance in ResNet+aug and PoseFormer+aug (both vanilla+aug), as well as GotenNet hybrid+aug, produces more accurate 3D poses than exact equivariance (fully equivariant), or no equivariance (vanilla). All 3D estimation results are captured from the same viewing angle.