Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion

TL;DR

This work introduces LA2DS, a 2D skeleton heatmap–driven, self-supervised temporal video alignment framework that leverages a video transformer to attend across space and time. By converting 2D poses into heatmaps and applying lightweight heatmap augmentations, the method achieves higher accuracy and greater robustness to missing or noisy keypoints than 3D-skeleton–based approaches like CASA. The approach is extended with a multi-modality RGB fusion that delivers state-of-the-art results on Penn Action, IKEA ASM, and H2O for various fine-grained human activity tasks, including phase classification, progression, and frame retrieval. The work also demonstrates the benefits of combining 2D heatmaps with RGB data, establishing a privacy-preserving, high-performance framework for temporal video alignment and fine-grained action understanding.

Abstract

This paper presents a self-supervised temporal video alignment framework which is useful for several fine-grained human activity understanding applications. In contrast with the state-of-the-art method of CASA, where sequences of 3D skeleton coordinates are taken directly as input, our key idea is to use sequences of 2D skeleton heatmaps as input. Unlike CASA which performs self-attention in the temporal domain only, we feed 2D skeleton heatmaps to a video transformer which performs self-attention both in the spatial and temporal domains for extracting effective spatiotemporal and contextual features. In addition, we introduce simple heatmap augmentation techniques based on 2D skeletons for self-supervised learning. Despite the lack of 3D information, our approach achieves not only higher accuracy but also better robustness against missing and noisy keypoints than CASA. Furthermore, extensive evaluations on three public datasets, i.e., Penn Action, IKEA ASM, and H2O, demonstrate that our approach outperforms previous methods in different fine-grained human activity understanding tasks. Finally, fusing 2D skeleton heatmaps with RGB videos yields the state-of-the-art on all metrics and datasets. To our best knowledge, our work is the first to utilize 2D skeleton heatmap inputs and the first to explore multi-modality fusion for temporal video alignment. Our code and dataset are available on our research website: https://retrocausal.ai/research/.

Figures (12)

• Figure 1: (a) The state-of-the-art method in self-supervised learning, i.e., CASA, uses 3D skeleton-based temporal video alignment as a pretext task and performs 3D skeleton augmentations. (b) Our approach relies on 2D skeleton-based temporal video alignment and conducts 2D skeleton augmentations. We use 2D skeleton heatmaps, which are fed to a video transformer for learning useful spatiotemporal and contextual features. Our method obtains higher accuracy and better robustness against missing and noisy keypoints, while showing superior performance in various fine-grained human activity understanding tasks. (c) We further fuse 2D skeleton heatmaps with RGB videos, establishing the state-of-the-art across all metrics and datasets.
• Figure 2: During training, our approach takes as input sequences of original heatmaps and augmented heatmaps. We perform self-attention both in the spatial and temporal domains to extract effective spatiotemporal and contextual cues within each sequence and cross-attention to extract contextual cues across the sequences. The extracted features after projection heads are passed to a matching module to predict correspondences across the sequences. Matching labels generated by the augmentation module are used in the loss function.
• Figure 3: Examples of our heatmap augmentation techniques based on 2D skeletons. The original 2D skeleton is in pink, while the augmented 2D skeleton is in red.
• Figure 4: (a) CASA directly processes 3D skeleton coordinates and conducts self-attention in the temporal domain only. (b) Our approach operates on 2D skeleton heatmaps and performs self-attention both in the spatial and temporal domains.
• Figure 5: Example videos with diverse camera viewpoints, actors, backgrounds, and actions from Penn Action, IKEA ASM, and H2O.