Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition

TL;DR

This work tackles efficient multimodal skeleton-based action recognition by proposing a self-supervised Decomposition and Composition framework that leverages a two-stream spatial-temporal backbone. It decomposes fused multimodal features into unimodal components and composes unimodal features to supervise and enhance multimodal representations, augmented with viewpoint-invariant training. The training objective combines $L = \alpha L_d + \beta L_c + L_{reg}$, and extensive experiments on NTU-60, NTU-120, and PKU-MMD II demonstrate strong performance with lower compute than many late-fusion approaches, indicating strong transferability and practicality. Overall, the approach offers a scalable, privacy-friendly, and robust solution for multimodal skeleton action understanding with broad applicability to real-world scenarios.

Abstract

Multimodal human action understanding is a significant problem in computer vision, with the central challenge being the effective utilization of the complementarity among diverse modalities while maintaining model efficiency. However, most existing methods rely on simple late fusion to enhance performance, which results in substantial computational overhead. Although early fusion with a shared backbone for all modalities is efficient, it struggles to achieve excellent performance. To address the dilemma of balancing efficiency and effectiveness, we introduce a self-supervised multimodal skeleton-based action representation learning framework, named Decomposition and Composition. The Decomposition strategy meticulously decomposes the fused multimodal features into distinct unimodal features, subsequently aligning them with their respective ground truth unimodal counterparts. On the other hand, the Composition strategy integrates multiple unimodal features, leveraging them as self-supervised guidance to enhance the learning of multimodal representations. Extensive experiments on the NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II datasets demonstrate that the proposed method strikes an excellent balance between computational cost and model performance.

Figures (4)

• Figure 1: Different multimodal fusion strategies. Late fusion involves the fusion of predicted probabilities or encoded features generated by individual models. Early fusion, on the other hand, entails the fusion directly from raw data, whereas embedding fusion pertains to fusion within the embedded feature space. It should be noted that in this work, early fusion refers to the fusion that occurs at the data level.
• Figure 2: The overall pipeline of the proposed method. We use different colors for the skeletons and features to distinguish different input streams of single and unified modalities. In the left part of the figure, it shows that the spatial-temporal encoders of the two branches encode the input separately to obtain decoupled features. The multimodal features are generated using fused embeddings before the encoder. Unimodal Feature Decomposition (UFD) aims to decompose features for more refined comparisons. Multimodal Feature Composition (MFC) aims to build up late fused multimodal features to enhance the features from multimodal embeddings.
• Figure 3: (a) Impact of the inter-modal consistency loss and multi-view training. (b) Comparison of action recognition results on NTU-60 of x-sub protocol with different features.
• Figure 4: Comparisons of learned features from 10 randomly selected different categories between the baseline and ours.