Curriculum-Based Strategies for Efficient Cross-Domain Action Recognition

TL;DR

This work tackles cross-view action recognition under a challenging domain shift from ground- to aerial-view perspectives. It shows that curriculum-based training using synthetic aerial data and real ground data—without real aerial data in training—can achieve competitive accuracy to naïve data mixing while substantially reducing training iterations. Two curriculum strategies are explored: a two-step fine-tuning approach and a multi-step progressive learning schedule, both validated on REMAG with SlowFast and MViTv2. The transformer-based MViTv2 generally outperforms the CNN-based SlowFast, and the progressive curriculum provides the largest efficiency gains. Overall, curriculum-based training offers a practical path to efficient, robust cross-domain action recognition in settings with limited aerial data availability.

Abstract

Despite significant progress in human action recognition, generalizing to diverse viewpoints remains a challenge. Most existing datasets are captured from ground-level perspectives, and models trained on them often struggle to transfer to drastically different domains such as aerial views. This paper examines how curriculum-based training strategies can improve generalization to unseen real aerial-view data without using any real aerial data during training. We explore curriculum learning for cross-view action recognition using two out-of-domain sources: synthetic aerial-view data and real ground-view data. Our results on the evaluation on order of training (fine-tuning on synthetic aerial data vs. real ground data) shows that fine-tuning on real ground data but differ in how they transition from synthetic to real. The first uses a two-stage curriculum with direct fine-tuning, while the second applies a progressive curriculum that expands the dataset in multiple stages before fine-tuning. We evaluate both methods on the REMAG dataset using SlowFast (CNN-based) and MViTv2 (Transformer-based) architectures. Results show that combining the two out-of-domain datasets clearly outperforms training on a single domain, whether real ground-view or synthetic aerial-view. Both curriculum strategies match the top-1 accuracy of simple dataset combination while offering efficiency gains. With the two-step fine-tuning method, SlowFast achieves up to a 37% reduction in iterations and MViTv2 up to a 30% reduction compared to simple combination. The multi-step progressive approach further reduces iterations, by up to 9% for SlowFast and 30% for MViTv2, relative to the two-step method. These findings demonstrate that curriculum-based training can maintain comparable performance (top-1 accuracy within 3% range) while improving training efficiency in cross-view action recognition.

Figures (6)

• Figure 1: Curriculum-based learning strategies for cross-view domain transfer. We compare three approaches for training action recognition models using two out-of-domain datasets: synthetic aerial-view and real ground-view videos. All models are evaluated on the unseen domain of real aerial-view videos. In this plot, we report the performance of the different methods on MViTv2 as the top-1 accuracy and the efficiency as the number of iterations (we fix all other parameters including batch size, input resolution, augmentations, and optimizer). Our results demonstrate that while all methods yield comparable top-1 accuracy, the progressive strategy achieves the highest training efficiency.
• Figure 2: Overview of our progressive learning pipeline. (a) Viewpoint–domain matrix of training and testing modalities. We use synthetic aerial and real ground-view data for training, and evaluate only on real aerial-view data. (b) Progressive learning procedure with $R=3$ rounds of training. The model is updated iteratively across rounds. The first $R{-}1$ rounds are trained with synthetic aerial data. In the final round, we compare two variants: (i) combined training using both synthetic aerial and real ground-view data, and (ii) fine-tuning using only real ground-view data.
• Figure 3: Top-1 accuracy across training strategies for two test sets and two model architectures (SlowFast and MViTv2). Each group corresponds to a training method: Real Only, Synthetic Only, Real (G) + Synthetic (A), Non-Progressive + FT (R-to-S), Non-Progressive + FT (S-to-R), and Progressive + FT. Bars are grouped by model type within each test set. Results show that combining real and synthetic data—especially through structured curricula—significantly improves generalization to the target aerial-view domain.
• Figure 4: Evaluation of computational efficiency across training strategies. We compare the total number of training iterations required for each method—Real (G) + Synthetic (A), Non-Progressive + FT (S-to-R), and Progressive + FT—across two test sets for both SlowFast and MViTv2. While all methods achieve comparable top-1 accuracy, the Progressive + FT approach consistently requires fewer iterations, highlighting its efficiency. Notably, Progressive + FT reduces the training cost by up to 30% compared to naive or two-step fine-tuning strategies, making it a favorable option in resource-constrained settings. Since the batch size, input resolution, augmentations, and optimizer are fixed, the per-iteration cost is effectively constant, making iteration count a reliable proxy for wall-clock training time within a model.
• Figure 5: Confusion Matrix 1 -- We show the detailed results for SlowFast and MViTv2. We observe the results for Real Aerial Only, Real Ground Only, and Synthetic Aerial Only.