LAC: Latent Action Composition for Skeleton-based Action Segmentation

TL;DR

Latent Action Composition (LAC) addresses skeleton-based action segmentation in untrimmed videos by learning from synthesized composable motions. It introduces Linear Action Decomposition ($oldsymbol{D}_v$) to represent high-level motions as an orthogonal basis, enabling linear combination of latent Motion and Static components to generate new sequences. A contrastive learning framework trains a skeleton visual encoder in both video-space and frame-space, offering end-to-end transfer to segmentation without requiring separate temporal models. Pre-training on Posetics and subsequent transfer to TSU, Charades, and PKU-MMD yields state-of-the-art results, demonstrating strong generalization and data efficiency; the approach can be extended to RGB data for further gains. Training the generator uses $L_{gen} = L_{self} + L_{target}$ to enforce reconstruction and cross-reconstruction of composable motions.

Abstract

Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC), a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthesized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.

Figures (4)

• Figure 1: General pipeline of LAC. Firstly, in the representation learning stage (left), we propose (i) a novel action generation module to combine skeletons of multiple videos (e.g., 'Walking' and 'Drinking' shown in the top and bottom respectively). We then adopt a (ii) contrastive module to pre-train a visual encoder by learning data augmentation invariant representations of the generated skeletons in both video space and frame space. Secondly (right), the pre-trained visual encoder is evaluated by transferring to action segmentation tasks.
• Figure 2: Overview of the Composable Action Generation model in LAC. The model consists of a visual encoder $\mathop{\mathrm{E_{LAC}}}\nolimits$ and a decoder $\mathop{\mathrm{D_{LAC}}}\nolimits$. In the latent space, we apply Linear Action Decomposition (LAD) by learning a visual action dictionary $\mathbf{D}_v$, which is an orthogonal basis where each vector represents a basic 'Motion'/'Static' transformation. Given a pair of skeleton sequences $\mathbf{p}_{m,c}$ and $\mathbf{p}_{m',c'}$, (i) their latent codes $\mathbf{r}_{m, c}$ and $\mathbf{r}_{m',c'}$ are embedded by $\mathop{\mathrm{E_{LAC}}}\nolimits$. (ii) Their projections $A_m$, $A_c$ and $A_{m'}$, $A_{c'}$ along $\mathbf{D}_v$ can be computed. The linear combination of $A_m$/$A_{m'}$ with corresponding directions in $\mathbf{D}_v$ constitutes the 'Motion' features and similarly the 'Static' features can also be obtained. (iii) In the training stage, we leverage motion retargeting for learning the whole framework by swapping their 'Motion' features and generating transferred motions. (iv) In the inference stage, we adopt linear combination of $\mathbf{r}_{m}$ and $\mathbf{r}_{m'}$ to obtain the composable motion features $\mathbf{r}_{mm'}$ and the composable skeleton sequences can be generated.
• Figure 3: Motion composition visualization. The input pair of videos and corresponding skeleton sequences (left) have simple motions. The generated skeleton sequences (right) are composed by both motions while keeping their respective viewpoint and body size ('Static') invariant.
• Figure 4: Linear manipulation of six 'Motion' directions in $\mathbf{D}_v$ on a skeleton sequence. Results indicate that each direction represents a meaningful motion transformation from a 'reference pose' marked in red (e.g., $\mathbf{d_m}_8$ for squat, $\mathbf{d_m}_{32}$ for bending over).