FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment

TL;DR

FineParser tackles the challenge of action quality assessment by learning fine-grained spatio-temporal representations of human-centric foreground actions. It introduces four components—Spatial Action Parser (SAP), Temporal Action Parser (TAP), Static Visual Encoder (SVE), and Fine-grained Contrastive Regression (FineReg)—to align actions across time and space and to quantify step-wise quality differences. The approach is complemented by FineDiving-HM, a dense mask annotation dataset that highlights foreground action regions to enhance credibility and interpretability. Empirical results on FineDiving-HM demonstrate state-of-the-art AQA performance and improved temporal parsing, suggesting practical benefits for real-world, credibility-sensitive sports analysis.

Abstract

Existing action quality assessment (AQA) methods mainly learn deep representations at the video level for scoring diverse actions. Due to the lack of a fine-grained understanding of actions in videos, they harshly suffer from low credibility and interpretability, thus insufficient for stringent applications, such as Olympic diving events. We argue that a fine-grained understanding of actions requires the model to perceive and parse actions in both time and space, which is also the key to the credibility and interpretability of the AQA technique. Based on this insight, we propose a new fine-grained spatial-temporal action parser named \textbf{FineParser}. It learns human-centric foreground action representations by focusing on target action regions within each frame and exploiting their fine-grained alignments in time and space to minimize the impact of invalid backgrounds during the assessment. In addition, we construct fine-grained annotations of human-centric foreground action masks for the FineDiving dataset, called \textbf{FineDiving-HM}. With refined annotations on diverse target action procedures, FineDiving-HM can promote the development of real-world AQA systems. Through extensive experiments, we demonstrate the effectiveness of FineParser, which outperforms state-of-the-art methods while supporting more tasks of fine-grained action understanding. Data and code are available at \url{https://github.com/PKU-ICST-MIPL/FineParser_CVPR2024}.

Figures (5)

• Figure 1: An overview of fine-grained spatial-temporal action parser (FineParser). It enhances human-centric foreground action representations by exploiting fine-grained semantic consistency and spatial-temporal correlation between video frames, improving the AQA performance. Green, red, yellow, and blue dashed lines represent the fine-grained alignment of target actions between query and exemplar videos in time and space within the same semantics.
• Figure 2: The architecture of the proposed FineParser. Given a pair of query and exemplar videos, spatial action parser (SAP) and temporal action parser (TAP) extract spatial-temporal representations of human-centric foreground actions in pairwise videos, as well as predict both target action masks and step transitions. The static visual encoder (SVE) captures static visual representations combined with the target action representation to mine more contextual details. Finally, fine-grained contrastive regression (FineReg) utilizes the representations to predict the action score of the query video.
• Figure 3: Examples of human-centric action mask annotations for the FineDiving dataset. The right line indicates the action type.
• Figure 4: The distribution of human-centric foreground action masks. The largest number of mask instances is 35,287, belonging to the action type 107B. The smallest number of mask instances is 101, containing the action types 109B, 201A, 201C, and 303C.
• Figure 5: Visualization of the predictions of target action masks produced by SAP. The predicted masks can focus on the target action regions in each frame, minimizing the impact of invalid backgrounds on action quality assessment.