Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BBoxMaskPose v2: Expanding Mutual Conditioning to 3D

Miroslav Purkrabek, Constantin Kolomiiets, Jiri Matas

TL;DR

The paper introduces BMPv2, an enhanced framework for multi-person pose estimation that combines PMPose, a probabilistic, mask-conditioned 2D pose estimator, with SAM-pose2seg, a pose-guided segmentation refinement. This mutual conditioning yields state-of-the-art results on standard COCO data and crowded OCHuman scenes, including surpassing 50 AP on OCHuman and significantly improving instance segmentation. A key insight is that stronger 2D pose quality directly benefits 3D pose estimation when prompted by segmentation masks, demonstrated via SAM-3D-Body prompts. To better evaluate real-world crowded scenes, the authors release OCHuman-Pose, an expanded annotation set addressing missing instances. The work also discusses a high-cost BMPv2+ variant that loops PMPose and SAM-pose2seg to maximize pose accuracy in exchange for computation, and shows domain-shift robustness on unseen data such as infant imagery.

Abstract

Most 2D human pose estimation benchmarks are nearly saturated, with the exception of crowded scenes. We introduce PMPose, a top-down 2D pose estimator that incorporates the probabilistic formulation and the mask-conditioning. PMPose improves crowded pose estimation without sacrificing performance on standard scenes. Building on this, we present BBoxMaskPose v2 (BMPv2) integrating PMPose and an enhanced SAM-based mask refinement module. BMPv2 surpasses state-of-the-art by 1.5 average precision (AP) points on COCO and 6 AP points on OCHuman, becoming the first method to exceed 50 AP on OCHuman. We demonstrate that BMP's 2D prompting of 3D model improves 3D pose estimation in crowded scenes and that advances in 2D pose quality directly benefit 3D estimation. Results on the new OCHuman-Pose dataset show that multi-person performance is more affected by pose prediction accuracy than by detection. The code, models, and data are available on https://MiraPurkrabek.github.io/BBox-Mask-Pose/.

BBoxMaskPose v2: Expanding Mutual Conditioning to 3D

TL;DR

The paper introduces BMPv2, an enhanced framework for multi-person pose estimation that combines PMPose, a probabilistic, mask-conditioned 2D pose estimator, with SAM-pose2seg, a pose-guided segmentation refinement. This mutual conditioning yields state-of-the-art results on standard COCO data and crowded OCHuman scenes, including surpassing 50 AP on OCHuman and significantly improving instance segmentation. A key insight is that stronger 2D pose quality directly benefits 3D pose estimation when prompted by segmentation masks, demonstrated via SAM-3D-Body prompts. To better evaluate real-world crowded scenes, the authors release OCHuman-Pose, an expanded annotation set addressing missing instances. The work also discusses a high-cost BMPv2+ variant that loops PMPose and SAM-pose2seg to maximize pose accuracy in exchange for computation, and shows domain-shift robustness on unseen data such as infant imagery.

Abstract

Most 2D human pose estimation benchmarks are nearly saturated, with the exception of crowded scenes. We introduce PMPose, a top-down 2D pose estimator that incorporates the probabilistic formulation and the mask-conditioning. PMPose improves crowded pose estimation without sacrificing performance on standard scenes. Building on this, we present BBoxMaskPose v2 (BMPv2) integrating PMPose and an enhanced SAM-based mask refinement module. BMPv2 surpasses state-of-the-art by 1.5 average precision (AP) points on COCO and 6 AP points on OCHuman, becoming the first method to exceed 50 AP on OCHuman. We demonstrate that BMP's 2D prompting of 3D model improves 3D pose estimation in crowded scenes and that advances in 2D pose quality directly benefit 3D estimation. Results on the new OCHuman-Pose dataset show that multi-person performance is more affected by pose prediction accuracy than by detection. The code, models, and data are available on https://MiraPurkrabek.github.io/BBox-Mask-Pose/.
Paper Structure (19 sections, 5 figures, 8 tables)

This paper contains 19 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. PMPose
  5. SAM-pose2seg
  6. Adapting SAM for pose2seg task
  7. BMP+
  8. From 2D to 3D
  9. Data
  10. OCHuman 2.0
  11. CIHP
  12. Experiments
  13. 2D Pose Estimation
  14. Detection and Segmentation
  15. Ablation Study
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Comparison of PMPose with state-of-the-art models on the COCO and the OCHuman datasets. The bubble diameter is proportional to a logarithm of number of model parameters. PMPose generalizes the best to unseen OCHuman dataset while keeping SOTA level on COCO.
  • Figure 2: Instance density measured by $\textit{IoU}_{\textit{max}}$ in OCHuman, OCHuman-Pose and CIHP datasets. Despite having almost the same mean of $\textit{IoU}_{\textit{max}}$ as OCHuman (\ref{['tab:ochuman-original-stats']}), OCHuman-Pose has a more realistic distribution of in-the-wild images, free from artificial instance selections. CIHP has different distribution due to focusing on crowds instead of human interactions.
  • Figure 3: Missing annotations in OCHuman. The originally annotated instances are often in the background, which causes problems in COCO-style AP evaluation. OCHuman-Pose expands the original dataset with new instances and previously ignored annotations.
  • Figure 4: BMPv2 results on a non-public dataset from infants. It is able to correctly disentangle individuals even in an unseen domain. Thanks to PMPose, keypoints outside of the image are not predicted inside. Results are shown only for authorized scenes; the dataset itself is more diverse than these examples. Infants anonymized with BLANKET blanket.
  • Figure 5: Additional qualitative results from OCHuman.