Localization-Aware Multi-Scale Representation Learning for Repetitive Action Counting

TL;DR

The paper tackles counting repetitive actions in realistic videos where interruptions and variable action durations degrade performance. It introduces Localization-Aware Multi-Scale Representation Learning (LMRL), featuring the Multi-Scale Period-Aware Representation (MPR) and Repetition Foreground Localization (RFL) branches that are trained jointly to produce discriminative periodic representations. Key contributions include a scale-specific similarity mechanism, foreground-background localization supervision, and a density-map based period predictor, all validated on RepCountA and UCFRep where LMRL outperforms prior methods. The work advances RAC robustness and suggests practical applications in domains like kitchen activity recognition and continuous human-action monitoring.

Abstract

Repetitive action counting (RAC) aims to estimate the number of class-agnostic action occurrences in a video without exemplars. Most current RAC methods rely on a raw frame-to-frame similarity representation for period prediction. However, this approach can be significantly disrupted by common noise such as action interruptions and inconsistencies, leading to sub-optimal counting performance in realistic scenarios. In this paper, we introduce a foreground localization optimization objective into similarity representation learning to obtain more robust and efficient video features. We propose a Localization-Aware Multi-Scale Representation Learning (LMRL) framework. Specifically, we apply a Multi-Scale Period-Aware Representation (MPR) with a scale-specific design to accommodate various action frequencies and learn more flexible temporal correlations. Furthermore, we introduce the Repetition Foreground Localization (RFL) method, which enhances the representation by coarsely identifying periodic actions and incorporating global semantic information. These two modules can be jointly optimized, resulting in a more discerning periodic action representation. Our approach significantly reduces the impact of noise, thereby improving counting accuracy. Additionally, the framework is designed to be scalable and adaptable to different types of video content. Experimental results on the RepCountA and UCFRep datasets demonstrate that our proposed method effectively handles repetitive action counting.

Figures (4)

• Figure 1: Two challenging cases existing in Repetitive Action Counting (face masked). (a) A man squats slower as physical strength declines. The action duration often changes, leading to failure prediction in previous methods due to their single-scale representation. In contrast, we propose a new Localization-Aware-Multi-Scale Representation Learning approach to handle frequency variation. (b) A woman rests between exercise sets. Our proposed methods refine the representation by localizing the foreground interests, shown as continuous repetitive exercising action blocks.
• Figure 2: Multi-Scale Period-Aware Representation.
• Figure 3: Detailed RFL branch.
• Figure 4: Visualization of density map prediction. The intensity of red color indicates the level of action density.