Online Action Representation using Change Detection and Symbolic Programming

TL;DR

The paper tackles online action representation by combining online change detection with symbolic motion primitives to segment streaming actions and encode them in a high-level, part-based form. It leverages CuSum change detection on joint angle and bone-length signals to determine action boundaries and then fits each segment's keypoint trajectories to simple primitives, capturing start, mid, and end dynamics. This representation enables online class-agnostic repetition counting through consecutive primitive matches, and it demonstrates competitive performance against offline methods while maintaining interpretability and compositionality. The approach offers practical value for online rehabilitation, surveillance, and activity analysis where future frames are unavailable and data may be sparse.

Abstract

This paper addresses the critical need for online action representation, which is essential for various applications like rehabilitation, surveillance, etc. The task can be defined as representation of actions as soon as they happen in a streaming video without access to video frames in the future. Most of the existing methods use predefined window sizes for video segments, which is a restrictive assumption on the dynamics. The proposed method employs a change detection algorithm to automatically segment action sequences, which form meaningful sub-actions and subsequently fit symbolic generative motion programs to the clipped segments. We determine the start time and end time of segments using change detection followed by a piece-wise linear fit algorithm on joint angle and bone length sequences. Domain-specific symbolic primitives are fit to pose keypoint trajectories of those extracted segments in order to obtain a higher level semantic representation. Since this representation is part-based, it is complementary to the compositional nature of human actions, i.e., a complex activity can be broken down into elementary sub-actions. We show the effectiveness of this representation in the downstream task of class agnostic repetition detection. We propose a repetition counting algorithm based on consecutive similarity matching of primitives, which can do online repetition counting. We also compare the results with a similar but offline repetition counting algorithm. The results of the experiments demonstrate that, despite operating online, the proposed method performs better or on par with the existing method.

Figures (9)

• Figure 1: (a) Joint angle sequences over one instance of jumping jack (b) Illustration of primitives fit to each keypoint
• Figure 2: Action representation pipeline
• Figure 3: The signal represents the variation of left thigh bone length(indicated in red) as a knee raise is performed. First, a meaningful segment is trimmed out using change detection, followed by fitting a primitive to each trajectory of the keypoints within that segment. In the figure, a representative fit to one of the keypoint(left knee) is indicated.
• Figure 4: In a jumping jack instance, a change was detected in the left shoulder joint angle in the upward motion.
• Figure 5: Once a match is found in the buffer, consecutive matches are searched for by drawing more primitives, and this continues until the match chain is broken. Groups of consecutive matches are loops.