Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skarimva: Skeleton-based Action Recognition is a Multi-view Application

Daniel Bermuth, Alexander Poeppel, Wolfgang Reif

TL;DR

By making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly, suggesting that the quality of the input data is currently a limiting factor for the performance of these models.

Abstract

Human action recognition plays an important role when developing intelligent interactions between humans and machines. While there is a lot of active research on improving the machine learning algorithms for skeleton-based action recognition, not much attention has been given to the quality of the input skeleton data itself. This work demonstrates that by making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly. This suggests that the quality of the input data is currently a limiting factor for the performance of these models. Based on these results, it is argued that the cost-benefit ratio of using multiple cameras is very favorable in most practical use-cases, therefore future research in skeleton-based action recognition should consider multi-view applications as the standard setup.

Skarimva: Skeleton-based Action Recognition is a Multi-view Application

TL;DR

By making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly, suggesting that the quality of the input data is currently a limiting factor for the performance of these models.

Abstract

Human action recognition plays an important role when developing intelligent interactions between humans and machines. While there is a lot of active research on improving the machine learning algorithms for skeleton-based action recognition, not much attention has been given to the quality of the input skeleton data itself. This work demonstrates that by making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly. This suggests that the quality of the input data is currently a limiting factor for the performance of these models. Based on these results, it is argued that the cost-benefit ratio of using multiple cameras is very favorable in most practical use-cases, therefore future research in skeleton-based action recognition should consider multi-view applications as the standard setup.
Paper Structure (16 sections, 4 figures, 10 tables)

This paper contains 16 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Improving Pose Estimates
  4. Results
  5. Improvements with new whole-body poses
  6. Comparison with other skeleton recalculations
  7. With multi-stream ensembling
  8. Few shot learning
  9. Real-time Applications
  10. Discussion
  11. Conclusion
  12. Calibration Details
  13. Real-time Applications
  14. Ensembling Modalities
  15. About objects and image inputs
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Example of a kick other person action with the new multi-view whole-body skeletons.
  • Figure 2: Extrinsic calibration by overlapping skeletons from the different views. The overlap can not be perfect due to the errors in the original pose estimations.
  • Figure 3: Confusion matrix of the ensembled ProtoGCN model on NTU-RGBD-60-xsub.
  • Figure 4: Confusion matrix of the ensembled ProtoGCN model on NTU-RGBD-120-xsub.