Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Event-based Human Pose Estimation with 3D Event Representations

Xiaoting Yin, Hao Shi, Jiaan Chen, Ze Wang, Yaozu Ye, Kailun Yang, Kaiwei Wang

TL;DR

Two 3D event representations are introduced: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV) and a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes is developed and released.

Abstract

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.

Exploring Event-based Human Pose Estimation with 3D Event Representations

TL;DR

Two 3D event representations are introduced: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV) and a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes is developed and released.

Abstract

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.
Paper Structure (2 sections, 6 figures, 1 table)

This paper contains 2 sections, 6 figures, 1 table.

Table of Contents

  1. Limitations
  2. Conclusion

Figures (6)

  • Figure 9: Qualitative comparison of different methods in outdoor gray-scale image sequences and event streams captured by our event camera. The human bounding boxes are estimated by the pre-trained YOLOv3 model redmon2018yolov3 using MMDetection chen2019mmdetection. Our two 3D event representation methods yield reliable estimates in street scenes and basements, which means stronger generalization ability in the real world.
  • Figure 10: (a) Our outdoor mobile robot is equipped with an event camera (DAVIS-346) and a laptop. (b) Event camera for capturing outdoor aligned grayscale frames and event information.
  • Figure 11: Visualization of feature maps and key point estimation results.
  • Figure 12: 2D results visualization on the DHP19 test dataset for different models (yellow for ground truth, blue for prediction).
  • Figure 13: Additional results by DHP19 calabrese2019dhp19, our PointNet qi2017pointnet, Pose-ResNet18† xiao2018simbase, and our DEV-Pose (ResNet18) on the DHP19 test dataset.
  • ...and 1 more figures