MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessment

TL;DR

Manifold-Aligned Graph Regularization (MAGR) is proposed, which first aligns deviated old features to the current feature manifold, ensuring representation consistency, and then constructs a graph jointly arranging old and new features aligned with quality scores.

Abstract

Action Quality Assessment (AQA) evaluates diverse skills but models struggle with non-stationary data. We propose Continual AQA (CAQA) to refine models using sparse new data. Feature replay preserves memory without storing raw inputs. However, the misalignment between static old features and the dynamically changing feature manifold causes severe catastrophic forgetting. To address this novel problem, we propose Manifold-Aligned Graph Regularization (MAGR), which first aligns deviated old features to the current feature manifold, ensuring representation consistency. It then constructs a graph jointly arranging old and new features aligned with quality scores. Experiments show MAGR outperforms recent strong baselines with up to 6.56%, 5.66%, 15.64%, and 9.05% correlation gains on the MTL-AQA, FineDiving, UNLV-Dive, and JDM-MSA split datasets, respectively. This validates MAGR for continual assessment challenges arising from non-stationary skill variations. Code is available at https://github.com/ZhouKanglei/MAGR_CAQA}{https://github.com/ZhouKanglei/MAGR_CAQA.

Figures (15)

• Figure 1: Traditional AQA vs CAQA: CAQA refines AQA from a few sequentially arrived instances without exhaustive retraining, which advances CL beyond classification.
• Figure 2: Our core idea: (a) Deviation of old features (blue circles) from the current manifold (orange curve) caused by the manifold shift; (b) Potential confusion for score regression due to the mixture of old features and current session features (green circles); (c) Translation of old features from the previous manifold (yellow curve) to the current; (d) Readjustment of the feature distribution to align with the quality score distribution.
• Figure 3: MAGR framework: (a) At the end of session $t-1$, representative samples are chosen and stored in the memory bank $\mathcal{M}^{t-1}$, and the feature extractor $f'$ is frozen. (b) Throughout session $t$, MP translates old features to the current manifold, while IIJ-GR regulates the entire feature space to align with the quality space. (c) After that, old features are first updated. (d) Then, new features are selected for the updated memory bank, denoted as $\mathcal{M}^{t}$, where the superscript indicates the update session.
• Figure 4: Illustrations of MP: Projector learning estimates the manifold shift, and feature projection translates old features to the current manifold.
• Figure 5: Illustrations of IIJ-GR: (a) Euclidean distance, (b) Angular distance, and (c) Distance Matrix Partitioning (DMP).