Enhancing 3D Human Pose Estimation Amidst Severe Occlusion with Dual Transformer Fusion

TL;DR

The paper tackles 3D Human Pose Estimation (HPE) from monocular video under severe occlusion, where missing 2D joints destabilize 3D lifting.It introduces Dual Transformer Fusion (DTF), which generates two intermediate views with a Multi-View Generator, refines them, and fuses them using cross-view attention, all guided by an Occlusion Guidance Mechanism that temporally interpolates missing joints and assigns confidence based on temporal distance.DTF is trained end-to-end using the mean per-joint position error loss $L_{MPJPE}$ and is evaluated on Human3.6M and MPI-INF-3DHP, where it achieves state-of-the-art performance under severe occlusion.The approach demonstrates robustness to sparse 2D detections and provides an occlusion-aware training protocol that can improve existing 2D-to-3D lifting methods, with potential extensions to multi-person scenarios.

Abstract

In the field of 3D Human Pose Estimation from monocular videos, the presence of diverse occlusion types presents a formidable challenge. Prior research has made progress by harnessing spatial and temporal cues to infer 3D poses from 2D joint observations. This paper introduces a Dual Transformer Fusion (DTF) algorithm, a novel approach to obtain a holistic 3D pose estimation, even in the presence of severe occlusions. Confronting the issue of occlusion-induced missing joint data, we propose a temporal interpolation-based occlusion guidance mechanism. To enable precise 3D Human Pose Estimation, our approach leverages the innovative DTF architecture, which first generates a pair of intermediate views. Each intermediate-view undergoes spatial refinement through a self-refinement schema. Subsequently, these intermediate-views are fused to yield the final 3D human pose estimation. The entire system is end-to-end trainable. Through extensive experiments conducted on the Human3.6M and MPI-INF-3DHP datasets, our method's performance is rigorously evaluated. Notably, our approach outperforms existing state-of-the-art methods on both datasets, yielding substantial improvements. The code is available here: https://github.com/MehwishG/DTF.

Figures (9)

• Figure 1: System diagram of the proposed DTF Algorithm: (a) Multi-View Generator (MVG), (b) Self-Refinement Module (SRM) (c) Combined Views Refinement (CVR), (d) Information Fusion Module (IFM), (e) Fused Views Refinement (FVR). Green markers show the available joints in the input sequence.
• Figure 2: Comparison of existing SOTA Methods including T3D-CNN ghafoor2022quantification, P-STMO shan2022p, MHFormer li2022mhformer, and STCFormer tang20233d with proposed DTF algorithm with a varying number of missing joints on MPI-INF-3DHP dataset. As the number of missing joints increases the proposed DTF algorithm has shown a minimum increase in error.
• Figure 3: Qualitative Comparison of existing SOTA methods with random $16$ missing joints per frame as input using Human3.6M dataset.
• Figure 4: (a) A sequence of seven detected 2D skeletons where the green marker shows the available joint in each frame. (b) 2D joint positions with confidence $(x,y,c)$, where (0,0,0) show missing joints. (c) Missing joint positions are interpolated and a confidence score is assigned based on the temporal gap.
• Figure 5: Histogram of Confidence Scores for SH-Detections in the Human3.6M Dataset.