Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition

Ruoqi Yin, Jianqin Yin

TL;DR

This work introduces THCT-Net, a two-stream hybrid CNN-Transformer network for skeleton-based human interaction recognition. The Transformer stream learns global inter-token dependencies using 3D-convolution embeddings and multi-head self-attention, while the CNN stream employs a multi-branch architecture that jointly learns spatio-temporal features from skeleton data, including a dual-branch pathway for raw coordinates and temporal differences. The two streams are fused via parallel splicing to capture multi-granularity context, with residual connections to speed training. Extensive experiments on NTU RGB+D 120 Mutual, H2O, and Assembly101 demonstrate state-of-the-art performance and strong cross-dataset generalization, validating the effectiveness of combining local and global representations for interactive actions.

Abstract

Human Interaction Recognition is the process of identifying interactive actions between multiple participants in a specific situation. The aim is to recognise the action interactions between multiple entities and their meaning. Many single Convolutional Neural Network has issues, such as the inability to capture global instance interaction features or difficulty in training, leading to ambiguity in action semantics. In addition, the computational complexity of the Transformer cannot be ignored, and its ability to capture local information and motion features in the image is poor. In this work, we propose a Two-stream Hybrid CNN-Transformer Network (THCT-Net), which exploits the local specificity of CNN and models global dependencies through the Transformer. CNN and Transformer simultaneously model the entity, time and space relationships between interactive entities respectively. Specifically, Transformer-based stream integrates 3D convolutions with multi-head self-attention to learn inter-token correlations; We propose a new multi-branch CNN framework for CNN-based streams that automatically learns joint spatio-temporal features from skeleton sequences. The convolutional layer independently learns the local features of each joint neighborhood and aggregates the features of all joints. And the raw skeleton coordinates as well as their temporal difference are integrated with a dual-branch paradigm to fuse the motion features of the skeleton. Besides, a residual structure is added to speed up training convergence. Finally, the recognition results of the two branches are fused using parallel splicing. Experimental results on diverse and challenging datasets, demonstrate that the proposed method can better comprehend and infer the meaning and context of various actions, outperforming state-of-the-art methods.

A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition

TL;DR

This work introduces THCT-Net, a two-stream hybrid CNN-Transformer network for skeleton-based human interaction recognition. The Transformer stream learns global inter-token dependencies using 3D-convolution embeddings and multi-head self-attention, while the CNN stream employs a multi-branch architecture that jointly learns spatio-temporal features from skeleton data, including a dual-branch pathway for raw coordinates and temporal differences. The two streams are fused via parallel splicing to capture multi-granularity context, with residual connections to speed training. Extensive experiments on NTU RGB+D 120 Mutual, H2O, and Assembly101 demonstrate state-of-the-art performance and strong cross-dataset generalization, validating the effectiveness of combining local and global representations for interactive actions.

Abstract

Human Interaction Recognition is the process of identifying interactive actions between multiple participants in a specific situation. The aim is to recognise the action interactions between multiple entities and their meaning. Many single Convolutional Neural Network has issues, such as the inability to capture global instance interaction features or difficulty in training, leading to ambiguity in action semantics. In addition, the computational complexity of the Transformer cannot be ignored, and its ability to capture local information and motion features in the image is poor. In this work, we propose a Two-stream Hybrid CNN-Transformer Network (THCT-Net), which exploits the local specificity of CNN and models global dependencies through the Transformer. CNN and Transformer simultaneously model the entity, time and space relationships between interactive entities respectively. Specifically, Transformer-based stream integrates 3D convolutions with multi-head self-attention to learn inter-token correlations; We propose a new multi-branch CNN framework for CNN-based streams that automatically learns joint spatio-temporal features from skeleton sequences. The convolutional layer independently learns the local features of each joint neighborhood and aggregates the features of all joints. And the raw skeleton coordinates as well as their temporal difference are integrated with a dual-branch paradigm to fuse the motion features of the skeleton. Besides, a residual structure is added to speed up training convergence. Finally, the recognition results of the two branches are fused using parallel splicing. Experimental results on diverse and challenging datasets, demonstrate that the proposed method can better comprehend and infer the meaning and context of various actions, outperforming state-of-the-art methods.
Paper Structure (12 sections, 7 equations, 2 figures, 1 table)

This paper contains 12 sections, 7 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. METHODOLOGY
  4. Transformer Stream
  5. CNN Stream
  6. EXPERIMENTS
  7. Datasets
  8. Implementation Details
  9. Results and Analyses
  10. Comparison with Baselines
  11. Comparison with Related Methods
  12. Conclusion

Figures (2)

  • Figure 1: Examples of individual actions (a), group activities (b) and interactive actions (c). (a) Pose of a single person raising his hand could depict the action Stretching. (b) Group activity Answering by raising hands is annotated regardless of the people. (c) In a scene of waving hello, each entity is an integral part of the interactive action.
  • Figure 2: The overall architecture of the proposed THCT-Net for skeleton-based human interaction recognition.