SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation

TL;DR

SMPLer addresses the quadratic cost of full attention in monocular 3D human shape and pose estimation by introducing decoupled attention and a compact SMPL-based target representation, enabling the use of high-resolution image features. It further enhances performance with multi-scale attention and a joint-aware attention module within a hierarchical Transformer, tightly integrating SMPL parameter estimation with 3D reconstruction. Empirically, SMPLer achieves $MPJPE = 45.2$ mm on Human3.6M, reducing error by over $10\%$ compared to Mesh Graphormer while using fewer than one-third the parameters, and attains real-time inference (approximately $96$ fps). The work demonstrates substantial efficiency and accuracy gains, and its SMPL-based output facilitates direct avatar control and downstream 3D applications, while offering a framework adaptable to other 3D reconstruction tasks.

Abstract

Existing Transformers for monocular 3D human shape and pose estimation typically have a quadratic computation and memory complexity with respect to the feature length, which hinders the exploitation of fine-grained information in high-resolution features that is beneficial for accurate reconstruction. In this work, we propose an SMPL-based Transformer framework (SMPLer) to address this issue. SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation, which allow effective utilization of high-resolution features in the Transformer. In addition, based on these two designs, we also introduce several novel modules including a multi-scale attention and a joint-aware attention to further boost the reconstruction performance. Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods both quantitatively and qualitatively. Notably, the proposed algorithm achieves an MPJPE of 45.2 mm on the Human3.6M dataset, improving upon Mesh Graphormer by more than 10% with fewer than one-third of the parameters. Code and pretrained models are available at https://github.com/xuxy09/SMPLer.

Figures (14)

• Figure 1: Two key designs of the proposed Transformer. The sub-caption "A-B" denotes the attention form "A" and the target representation "B", respectively. The vertical and horizontal lines around the rectangles represent query and key in the attention operation. Red indicates source image features, blue indicates target output representation, and the colors within the rectangles represent the interactions between them. (a) Existing Transformers for 3D human reconstruction lin2021endlin2021mesh typically adopt a ViT-style full attention operation and a vertex-based target representation, hindering the utilization of high-resolution image features. In contrast, we propose a decoupled attention (b) and an SMPL-based target representation (c), which effectively address the above problem and improve reconstruction performance. $l_T$, $l_\mathcal{T}$, and $l_F$ are the lengths of the vertex-based embedding, SMPL-based embedding, and image features, respectively. The area of each rectangle denotes the computation and memory complexity of the attention operation. Please refer to Section \ref{['sec: efficient attention']} for mathematical explanations.
• Figure 2: Overview of the proposed framework. Given a monocular input image, we first use a CNN backbone sun2019deep to extract image features $\mathcal{F}$, which are fed into the Transformer to reconstruct the 3D human body. The main ingredients of this framework are 1) an efficient decoupled attention module in the Transformer (Section \ref{['sec: efficient attention']}), and 2) a compact target representation $\mathcal{T}$ based on parametric human model (Section \ref{['sec: compact representation']}). More detailed descriptions of the Transformer architecture are provided in Figure \ref{['fig: hierarchical']}.
• Figure 3: Hierarchical architecture of our Transformer. (a) shows an overview of the hierarchical architecture which corresponds to the "Transformer" in Figure \ref{['fig: pipeline']}. With the image features $\mathcal{F}$, we progressively refine the initial estimation $P^0$ with $B$ Transformer Blocks (Eq. \ref{['eq: refine']}). In (b), each Transformer Block consists of $U$ Transformer Units, and each Unit is formulated as $h_\text{final}$ in Eq. \ref{['eq:h_final']}. The module "$\mathcal{J}$-Reg" represents 2D joint regression from the 3D estimation results, corresponding to Eq. \ref{['eq:smpl']}-\ref{['eq:2D_joints']}.
• Figure 4: Jointly exploiting multi-scale features in the attention operation (see Eq. \ref{['eq:enhanced multi-scale attention']} for more explanations).
• Figure 5: Pooling-based multi-scale positional encoding. We learn the positional encoding only for the highest-resolution feature, and the encodings for other scales are generated by average pooling, such that similar spatial locations across different scales have similar positional embeddings.