Understanding Spatio-Temporal Relations in Human-Object Interaction using Pyramid Graph Convolutional Network

TL;DR

A novel Pyramid Graph Convolutional Network (PGCN), which employs a pyramidal encoder-decoder architecture consisting of an attention based graph convolution network and a temporal pyramid pooling module for downsampling and upsampling interaction sequence on the temporal axis, is proposed.

Abstract

Human activities recognition is an important task for an intelligent robot, especially in the field of human-robot collaboration, it requires not only the label of sub-activities but also the temporal structure of the activity. In order to automatically recognize both the label and the temporal structure in sequence of human-object interaction, we propose a novel Pyramid Graph Convolutional Network (PGCN), which employs a pyramidal encoder-decoder architecture consisting of an attention based graph convolution network and a temporal pyramid pooling module for downsampling and upsampling interaction sequence on the temporal axis, respectively. The system represents the 2D or 3D spatial relation of human and objects from the detection results in video data as a graph. To learn the human-object relations, a new attention graph convolutional network is trained to extract condensed information from the graph representation. To segment action into sub-actions, a novel temporal pyramid pooling module is proposed, which upsamples compressed features back to the original time scale and classifies actions per frame. We explore various attention layers, namely spatial attention, temporal attention and channel attention, and combine different upsampling decoders to test the performance on action recognition and segmentation. We evaluate our model on two challenging datasets in the field of human-object interaction recognition, i.e. Bimanual Actions and IKEA Assembly datasets. We demonstrate that our classifier significantly improves both framewise action recognition and segmentation, e.g., F1 micro and F1@50 scores on Bimanual Actions dataset are improved by $4.3\%$ and $8.5\%$ respectively.

Figures (6)

• Figure 1: The initial spatial relation graph: (a) Spatial graph with notes (blue) and edges (orange) on example of Bimanual Actions dataset dreher2019learning; (b) Initial inwards adjacent matrix with skeleton inward edges (orange block), empty human-objects (red blocks) and objects-objects edges (blue block).
• Figure 2: Illustration of the attention unit (orange region) in a spatial convolutional layer
• Figure 3: Framework of temporal pyramid pooling decoder with three input graph feature maps: $\mathbf{G}^4$, $\mathbf{G}^7$ and $\mathbf{G}^{10}$, where $TP\ i$ is temporal pooling block with output size $i$.
• Figure 4: Normalized confusion matrix for the top prediction of accumulative framewise classification correctness over all folds on Bimanual Actions dataset dreher2019learning.
• Figure 5: Comparison of the qualitative results on Bimanual Actions dataset dreher2019learning for a sawing example