Rethinking temporal self-similarity for repetitive action counting

TL;DR

This work tackles repetitive action counting in long videos by moving away from using a temporal self-similarity matrix (TSM) as a bottleneck. It introduces RACnet, which learns full-temporal-resolution frame embeddings and augments them with a temporal repetition constraint loss (tReCo) based on a generated reference TSM, coupled with an action-start predictor that counts repetitions from per-frame start probabilities. The proposed method achieves state-of-the-art results on RepCount, UCFRep, and Countix, demonstrating improved accuracy and robustness due to full-resolution processing and the self-similarity alignment loss. The approach offers a practical impact for real-world tasks requiring accurate counting of repetitive actions, such as rehabilitation, by avoiding information loss from TSM bottlenecks and leveraging end-to-end temporal structure.

Abstract

Counting repetitive actions in long untrimmed videos is a challenging task that has many applications such as rehabilitation. State-of-the-art methods predict action counts by first generating a temporal self-similarity matrix (TSM) from the sampled frames and then feeding the matrix to a predictor network. The self-similarity matrix, however, is not an optimal input to a network since it discards too much information from the frame-wise embeddings. We thus rethink how a TSM can be utilized for counting repetitive actions and propose a framework that learns embeddings and predicts action start probabilities at full temporal resolution. The number of repeated actions is then inferred from the action start probabilities. In contrast to current approaches that have the TSM as an intermediate representation, we propose a novel loss based on a generated reference TSM, which enforces that the self-similarity of the learned frame-wise embeddings is consistent with the self-similarity of repeated actions. The proposed framework achieves state-of-the-art results on three datasets, i.e., RepCount, UCFRep, and Countix.

Figures (6)

• Figure 1: While previous works use TSMs as intermediate representation, we learn a representation where the TSM is consistent with the ground-truth.
• Figure 2: RACnet architecture. Our approach consists of three modules: The Feature Encoder generates per-frame embeddings from full-resolution videos. It is pre-trained and frozen. The Temporal Repetition Constrain Module generates a temporal self-similarity matrix as an auxiliary task, where the temporal Repetition Constrain (tReCo) loss is proposed to enforce consistency between the self-similarities of the features and the repeated actions. The SMS-TCN network in the Action Start Predictor (ASP) generates per-frame action start probabilities and the sum-of-squared loss (SSE) is used for training. The number of repetitions in the input videos is calculated as the number of frames that correspond to an action start.
• Figure 3: Examples of generated reference temporal self-similarity matrices for three different videos. Left: The actions start later in the video. Middle: There is a long break until the actions are continued. Right: The actions have different durations.
• Figure 4: Visualizations of TSMs. (a) Example video. (b) TSM of RepNet. (c) Reference TSM. (d) TSM of our approach. Yellow indicates high similarity and blue stands for low similarity.
• Figure 5: Example of incorrect annotation. From top to bottom: several key frames, 1D PCA of feature embeddings, ground truth action start, predicted action start probabilities.