Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D Human Pose Perception from Egocentric Stereo Videos

Hiroyasu Akada, Jian Wang, Vladislav Golyanik, Christian Theobalt

TL;DR

The paper addresses the challenge of accurate 3D human pose estimation from egocentric stereo videos under severe self-occlusion by introducing a transformer-based framework that fuses 2D heatmaps, depth from a video-based 3D scene reconstruction, and video-dependent joint queries to capture temporal context. It proposes a multi-stage pipeline including 2D pose estimation, body segmentation, 3D scene reconstruction via SfM, and a DETR-style decoder that attends over depth and heatmap memories with depth padding to handle missing data. The authors introduce UnrealEgo2 and UnrealEgo-RW datasets and demonstrate substantial improvements over prior stereo egocentric methods on both synthetic and real-world data, with ablations validating the contribution of depth, padding masks, and query adaptation. The work also provides pre-trained models and datasets to foster further development in egocentric 3D vision, potentially enabling realistic avatar animation and XR applications.

Abstract

While head-mounted devices are becoming more compact, they provide egocentric views with significant self-occlusions of the device user. Hence, existing methods often fail to accurately estimate complex 3D poses from egocentric views. In this work, we propose a new transformer-based framework to improve egocentric stereo 3D human pose estimation, which leverages the scene information and temporal context of egocentric stereo videos. Specifically, we utilize 1) depth features from our 3D scene reconstruction module with uniformly sampled windows of egocentric stereo frames, and 2) human joint queries enhanced by temporal features of the video inputs. Our method is able to accurately estimate human poses even in challenging scenarios, such as crouching and sitting. Furthermore, we introduce two new benchmark datasets, i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a much larger number of egocentric stereo views with a wider variety of human motions than the existing datasets, allowing comprehensive evaluation of existing and upcoming methods. Our extensive experiments show that the proposed approach significantly outperforms previous methods. We will release UnrealEgo2, UnrealEgo-RW, and trained models on our project page.

3D Human Pose Perception from Egocentric Stereo Videos

TL;DR

The paper addresses the challenge of accurate 3D human pose estimation from egocentric stereo videos under severe self-occlusion by introducing a transformer-based framework that fuses 2D heatmaps, depth from a video-based 3D scene reconstruction, and video-dependent joint queries to capture temporal context. It proposes a multi-stage pipeline including 2D pose estimation, body segmentation, 3D scene reconstruction via SfM, and a DETR-style decoder that attends over depth and heatmap memories with depth padding to handle missing data. The authors introduce UnrealEgo2 and UnrealEgo-RW datasets and demonstrate substantial improvements over prior stereo egocentric methods on both synthetic and real-world data, with ablations validating the contribution of depth, padding masks, and query adaptation. The work also provides pre-trained models and datasets to foster further development in egocentric 3D vision, potentially enabling realistic avatar animation and XR applications.

Abstract

While head-mounted devices are becoming more compact, they provide egocentric views with significant self-occlusions of the device user. Hence, existing methods often fail to accurately estimate complex 3D poses from egocentric views. In this work, we propose a new transformer-based framework to improve egocentric stereo 3D human pose estimation, which leverages the scene information and temporal context of egocentric stereo videos. Specifically, we utilize 1) depth features from our 3D scene reconstruction module with uniformly sampled windows of egocentric stereo frames, and 2) human joint queries enhanced by temporal features of the video inputs. Our method is able to accurately estimate human poses even in challenging scenarios, such as crouching and sitting. Furthermore, we introduce two new benchmark datasets, i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a much larger number of egocentric stereo views with a wider variety of human motions than the existing datasets, allowing comprehensive evaluation of existing and upcoming methods. Our extensive experiments show that the proposed approach significantly outperforms previous methods. We will release UnrealEgo2, UnrealEgo-RW, and trained models on our project page.
Paper Structure (13 sections, 11 equations, 6 figures, 7 tables)

This paper contains 13 sections, 11 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mobile Device and Datasets
  4. Method
  5. 2D Pose Estimation
  6. Human Body Segmentation
  7. 3D Scene Reconstruction
  8. 3D Pose Estimation
  9. Experiments
  10. Datasets for Evaluation
  11. Training Details
  12. Evaluation
  13. Conclusion

Figures (6)

  • Figure 1: 3D human pose estimation results of our proposed method from egocentric stereo fisheye videos. Left: results on synthetic images; (a) reference RGB view of the scene; (b) 3D-to-2D pose re-projections, and (c) a 3D pose in a scene mesh reconstructed by our framework. Right: results on real-world images; (d) reference view; (e) 3D-to-2D pose re-projections; (f) a 3D pose in the reconstructed scene, and (g) 3D virtual character animation (possible future application of our method).
  • Figure 2: Our portable setup to acquire UnrealEgo-RW.
  • Figure 3: Overview of our framework. Our method takes egocentric stereo videos $\{\mathbf{I}^{t}_{\text{Left}}, \mathbf{I}^{t}_{\text{Right}}\}$ as inputs. We first apply the 2D module to obtain 2D joint heatmaps $\{\mathbf{H}^{t}_{\text{Left}}, \mathbf{H}^{t}_{\text{Right}}\}$ and video features $\{\mathbf{F}^{t}_{\text{Left}}, \mathbf{F}^{t}_{\text{Right}}\}$ (Sec. \ref{['subsec:2D_module']}). The heatmaps are used with input videos to create human body masks $\{{\mathbf{M}}^{t}_{\text{Left}}, {\mathbf{M}}^{t}_{\text{Right}} \}$ (Sec. \ref{['subsec:sam_module']}). Next, we use uniformly sampled windows of input frames and human body masks to reconstruct a 3D scene mesh (Sec. \ref{['subsec:scene_module']}). From the mesh, we generate depth maps $\{{\mathbf{D}}^{t}_{\text{Left}}, {\mathbf{D}}^{t}_{\text{Right}} \}$ and depth region masks $\{{\mathbf{R}}^{t}_{\text{Left}}, {\mathbf{R}}^{t}_{\text{Right}}\}$. Note that this diagram shows an example case of missing depth values for the second input frame. Lastly, the depth data, 2D joint heatmaps, video features, joint queries $q^{t}$ and the padding masks ${V}^{t}_{\text{Depth}}$ are processed in the 3D module to estimate 3D poses ${\mathbf{P}}^{t}$ (Sec. \ref{['subsec:3D_module']}).
  • Figure 5: Results of our framework and comparison methods on example sequences from UnrealEgo2 (above) and UnrealEgo-RW (below). Left: MPJPE curves. Right: Outputs of our method at frame 87 and 329 of the sequences, respectively. 3D pose estimation and ground truth are colored in red and green, respectively.
  • Figure : Stereo inputs Akada et al.hakada2022unrealego Baseline Ours Stereo inputs Akada et al.hakada2022unrealego Baseline Ours
  • ...and 1 more figures