DeforHMR: Vision Transformer with Deformable Cross-Attention for 3D Human Mesh Recovery

TL;DR

DeforHMR is introduced, a novel regression-based monocular HMR framework designed to enhance the prediction of human pose parameters using deformable attention transformers and introduces an new, effective paradigm for decoding local spatial information from large pretrained vision encoders in computer vision.

Abstract

Human Mesh Recovery (HMR) is an important yet challenging problem with applications across various domains including motion capture, augmented reality, and biomechanics. Accurately predicting human pose parameters from a single image remains a challenging 3D computer vision task. In this work, we introduce DeforHMR, a novel regression-based monocular HMR framework designed to enhance the prediction of human pose parameters using deformable attention transformers. DeforHMR leverages a novel query-agnostic deformable cross-attention mechanism within the transformer decoder to effectively regress the visual features extracted from a frozen pretrained vision transformer (ViT) encoder. The proposed deformable cross-attention mechanism allows the model to attend to relevant spatial features more flexibly and in a data-dependent manner. Equipped with a transformer decoder capable of spatially-nuanced attention, DeforHMR achieves state-of-the-art performance for single-frame regression-based methods on the widely used 3D HMR benchmarks 3DPW and RICH. By pushing the boundary on the field of 3D human mesh recovery through deformable attention, we introduce an new, effective paradigm for decoding local spatial information from large pretrained vision encoders in computer vision.

Figures (4)

• Figure 1: Full system architecture of DeforHMR. We dedicate a learnable query embedding for each of the 24 joint articulations and the body shape vector.
• Figure 2: Our proposed deformable cross-attention module. The offset-generating convolutional neural network takes the spatial features from the transformer encoder to generate sampling position offsets $\Delta P$. These offsets are added to the grid reference positions $P_{ref}$ to get our final sampling positions $P$. These sampling positions are used to $1)$ sample the input context via bilinear interpolation, which is then projected to keys and values for attention computation, as well as $2)$ sample the relative positional embeddings (RPE) to get our attention bias. These are combined in the standard multi-head cross-attention formulation with relative position biases to generate the output.
• Figure 3: Qualitative results on test set. We visualize the original image and the predicted mesh projected onto the original image for both HMR2.0 goel2023humans and DeforHMR. We highlight the inaccurate mesh regions outputted by HMR2.0 in red boxes and highlight the corresponding mesh region on DeforHMR's mesh output in green boxes. Upon visualizing HMR2.0 and our model's recovered meshes on four distinct scenarios across 3DPW vonMarcard2018 and RICH huang2022capturinginferringdensefullbody, we observe DeforHMR's significant improvements on HMR2.0's ability to recover accurate 3D human meshes. While HMR2.0 shows inaccurate feet & hand positions in all four rows as well as inaccurate orientation of the entire torso in the lowest row, DeforHMR consistently shows more accurate feet, hand, and torso positions.
• Figure 4: Deformable Attention visualizations for first (left) and last (right) layer of decoder. In each pair of images, we can see important body areas where DeforHMR focuses on in order to model these challenging poses.