On the Use of Hierarchical Vision Foundation Models for Low-Cost Human Mesh Recovery and Pose Estimation

TL;DR

The paper tackles the challenge of deploying accurate image-based human mesh recovery (HMR) and human pose estimation (HPE) under limited compute. It introduces truncated encoders that use only the first two or three stages of hierarchical vision foundation models (VFMs) such as Swin, GroupMixFormer, and VMamba, while keeping the ViTPose/HMR2.0 decoders intact. Across 27 hierarchical-VFM–based models and standard benchmarks, the authors show that these truncated encoders achieve performance comparable to full-stage models and offer superior efficiency relative to existing lightweight baselines. The work provides practical, open-source baselines that facilitate real-time or edge deployment of HMR and HPE systems and motivates broader adoption of hierarchical VFMs for efficient human-centric vision tasks.

Abstract

In this work, we aim to develop simple and efficient models for human mesh recovery (HMR) and its predecessor task, human pose estimation (HPE). State-of-the-art HMR methods, such as HMR2.0 and its successors, rely on large, non-hierarchical vision transformers as encoders, which are inherited from the corresponding HPE models like ViTPose. To establish baselines across varying computational budgets, we first construct three lightweight HMR2.0 variants by adapting the corresponding ViTPose models. In addition, we propose leveraging the early stages of hierarchical vision foundation models (VFMs), including Swin Transformer, GroupMixFormer, and VMamba, as encoders. This design is motivated by the observation that intermediate stages of hierarchical VFMs produce feature maps with resolutions comparable to or higher than those of non-hierarchical counterparts. We conduct a comprehensive evaluation of 27 hierarchical-VFM-based HMR and HPE models, demonstrating that using only the first two or three stages achieves performance on par with full-stage models. Moreover, we show that the resulting truncated models exhibit better trade-offs between accuracy and computational efficiency compared to existing lightweight alternatives. The source code is available at https://github.com/nttcom/TruncHierVFM.

Figures (10)

• Figure 1: (Left): 3D pose estimation results (PA-MPJPE; lower is better) of human mesh recovery (HMR) models on the Human3.6M dataset ionescu+2014tpami. (Right): 2D pose estimation results (AP; higher is better) of human pose estimation (HPE) models on the COCO-val dataset lin+2014eccv. Circle size reflects model size. Gray circles indicate existing methods, while colored circles represent our proposed models, which leverage the early stages of hierarchical vision foundation models (VFMs) as encoders. These results demonstrate that our models tend to offer a more favorable trade-off between performance and computational cost ( i.e., GFLOPs) compared to existing approaches.
• Figure 4: Qualitative results of hierarchical-VFM-based HPE models.
• Figure 5: Qualitative results of hierarchical-VFM-based HMR models.
• Figure : (a) ViTPose xu+2022neurips
• Figure : (a)