Latent-Info and Low-Dimensional Learning for Human Mesh Recovery and Parallel Optimization

TL;DR

The paper tackles the problem of accurate and efficient 3D human mesh recovery from video by addressing limb misalignment and insufficient local detail, while reducing the computational burden of attention-based interactions. It introduces a two-stage framework that first extracts hybrid latent information in the frequency domain via a latent information frequency domain extractor, then models mesh–pose interactions with low-dimensional modules (LCP and LSP) in a parallelized architecture (LDMP). Key contributions include the latent information frequency domain extractor, the LDMP with cross-branch dimensionality reduction, and a substantial reduction in MACs without sacrificing accuracy, demonstrated on 3DPW, Human3.6M, and MPI-INF-3DHP. The method yields robust performance in outdoor and complex scenes and offers practical benefits for real-time or near real-time applications in animation, virtual try-on, and related fields.

Abstract

Existing 3D human mesh recovery methods often fail to fully exploit the latent information (e.g., human motion, shape alignment), leading to issues with limb misalignment and insufficient local details in the reconstructed human mesh (especially in complex scenes). Furthermore, the performance improvement gained by modelling mesh vertices and pose node interactions using attention mechanisms comes at a high computational cost. To address these issues, we propose a two-stage network for human mesh recovery based on latent information and low dimensional learning. Specifically, the first stage of the network fully excavates global (e.g., the overall shape alignment) and local (e.g., textures, detail) information from the low and high-frequency components of image features and aggregates this information into a hybrid latent frequency domain feature. This strategy effectively extracts latent information. Subsequently, utilizing extracted hybrid latent frequency domain features collaborates to enhance 2D poses to 3D learning. In the second stage, with the assistance of hybrid latent features, we model the interaction learning between the rough 3D human mesh template and the 3D pose, optimizing the pose and shape of the human mesh. Unlike existing mesh pose interaction methods, we design a low-dimensional mesh pose interaction method through dimensionality reduction and parallel optimization that significantly reduces computational costs without sacrificing reconstruction accuracy. Extensive experimental results on large publicly available datasets indicate superiority compared to the most state-of-the-art.

Figures (5)

• Figure 1: Our method explores the latent information of image features in the frequency domain to enhance representation capabilities. Therefore, despite being in outdoor complex scenes, we can still reconstruct more accurate human mesh. In contrast, the SOTA method easily produces inaccurate human mesh, such as limb location shifts and insufficient local details.
• Figure 2: (a) is the framework of our method. Our framework aims to reconstruct the corresponding human mesh given a video sequence. It consists of two stages: latent information extraction and mesh pose interaction modeling. (b) is the architecture of the low dimensional mesh pose interaction method in the stage 2. (c) and (d) are the low-dimensional collaborative perception attention and the low-dimensional self-perception attention in (b), respectively.
• Figure 3: In outdoor complex motion scenes, our method can recover accurate human mesh compared to the SOTA method.
• Figure 4: Visual ablation experiments on whether to extract latent information in complex scenes with indoor chaotic backgrounds.
• Figure 5: For videos found online, our method can restore accurate and smooth human sequences.