Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models

TL;DR

S2I is introduced, a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions that enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning.

Abstract

Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct application to 3D human skeleton data remains challenging due to fundamental differences in data format. Moreover, the scarcity of large-scale skeleton datasets and the need to incorporate skeleton data into multi-modal action recognition without introducing additional model branches present significant research opportunities. To address these challenges, we introduce Skeleton-to-Image Encoding (S2I), a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions. This encoding enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning, effectively transferring rich visual-domain knowledge to skeleton analysis. While existing skeleton methods often design models tailored to specific, homogeneous skeleton formats, they overlook the structural heterogeneity that naturally arises from diverse data sources. In contrast, our S2I representation offers a unified image-like format that naturally accommodates heterogeneous skeleton data. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate the effectiveness and generalizability of our method for self-supervised skeleton representation learning, including under challenging cross-format evaluation settings.

Figures (7)

• Figure 1: The existing methods train the skeleton models directly, while the proposed method converts skeleton data into image-like data and then train with the pre-trained vision models.
• Figure 1: Ablation study of image pretrain, skeleton pretrain on NTU-60 C-sub.
• Figure 2: Illustration of the Skeleton-to-Image Encoding (S2I) process, which transforms skeleton sequences into image-like representations via joint partitioning, temporal stacking, and interpolation.
• Figure 3: Effect of mask ratio on NTU-60 C-sub.
• Figure 4: Comparison between the existing skeleton-specific pipeline and our proposed S2I-based pipeline for transfer learning across datasets. (a) Existing methods first align joint formats before pretraining and fine-tuning on target datasets, which may lead to information loss. (b) Our approach bypasses manual joint selection by encoding the raw skeleton sequence into an image via S2I, enabling unified processing across datasets using a vision model.