Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling

TL;DR

The paper tackles continual learning for Action Quality Assessment (Continual-AQA) by proposing a dual strategy: FSCAR to fuse prior task knowledge through memory-efficient rehearsal and AGSG to separate general versus task-specific action knowledge. By enforcing a task-consistent, score-discriminative feature distribution across tasks, the approach mitigates forgetting when learning new actions. Empirical results on AQA-7, BEST, and MTL-AQA demonstrate significant improvements over existing CL baselines and prior AQA methods, with robust ablations validating the contributions of FSCAR and AGSG. The work provides a practical pathway for deploying AQA systems capable of expanding to new actions without retraining from scratch.

Abstract

Action Quality Assessment (AQA) is a task that tries to answer how well an action is carried out. While remarkable progress has been achieved, existing works on AQA assume that all the training data are visible for training at one time, but do not enable continual learning on assessing new technical actions. In this work, we address such a Continual Learning problem in AQA (Continual-AQA), which urges a unified model to learn AQA tasks sequentially without forgetting. Our idea for modeling Continual-AQA is to sequentially learn a task-consistent score-discriminative feature distribution, in which the latent features express a strong correlation with the score labels regardless of the task or action types.From this perspective, we aim to mitigate the forgetting in Continual-AQA from two aspects. Firstly, to fuse the features of new and previous data into a score-discriminative distribution, a novel Feature-Score Correlation-Aware Rehearsal is proposed to store and reuse data from previous tasks with limited memory size. Secondly, an Action General-Specific Graph is developed to learn and decouple the action-general and action-specific knowledge so that the task-consistent score-discriminative features can be better extracted across various tasks. Extensive experiments are conducted to evaluate the contributions of proposed components. The comparisons with the existing continual learning methods additionally verify the effectiveness and versatility of our approach. Data and code are available at https://github.com/iSEE-Laboratory/Continual-AQA.

Figures (8)

• Figure 1: Catastrophic forgetting in Continual-AQA. We sequentially train an AQA model on two tasks and show the performance on the first task in scatter plot and Spearman’s rank correlation coefficient ($\rho$). A higher $\rho$ (i.e., more points close to the red lines) indicates better performance.
• Figure 2: Different learning targets of Conventional Continual Learning and Continual-AQA. The learning target of conventional Continual Learning (left) is to construct new decision boundaries so that intra-class features can be clustered and inter-class features can be separated. Differently, the learning target of Continual-AQA (right) is to learn a task-consistent score-discriminative feature distribution where the features strongly correlate with the score labels regardless of the task or action types.
• Figure 3: A pipeline of our method for Continual-AQA. Our framework contains two main components: Feature-Score Correlation-Aware Rehearsal(FSCAR) in sub-image (b) and Feature Extractor with Action General-Specific Graph(AGSG) in sub-image (a). Notably, when a new task comes, the model is initialized by the previously learned weights, and a new Action-Specific Graph will be defined for the new task before training. An intuitive example of Feature-Score co-Augmentation is shown in sub-image (c), where a feature with a darker color has a higher quality score. Arrows in different colors indicate different data streams. $\{1,...,t\}$ is the index of tasks and here we regard task $t$as the current task and task $t\text{-}1$as the nearest previous task. $D^t$ denotes all the training data in current task, and $\psi=\{\psi^1,...,\psi^{t-1}\}$ denotes the stored exemplars of previous tasks. Best viewed in color.
• Figure 4: Feature extractor with the proposed Action General-Specific Graph (AGSG). Different from the original JRG JRG, in this work, we decouple the original joint relation graph into an Action-General Graph and an Action-Specific Graph to learn action-general and action-specific knowledge, respectively (the lower part).
• Figure 5: T-SNE Visualizations of the features sampled by Herding (a), Random Sampling (b) and our GS (c) after training on the first task. Points in greens denote the features. Darker color indicates higher score. Points in orange, blue and red denote the feature sampled by Herding, Random Sampling and our GS, respectively. The exemplars sampled by our GS can cover all score levels and roughly describe the learned feature distribution rather than gathering in a compact region. Best viewed in color.