Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training

TL;DR

UNITE tackles unsupervised video domain adaptation for action recognition by transferring from a labeled source domain to an unlabeled target domain, formalized with distributions $\\$\mathcal{P}_S$ and $\\$\mathcal{P}_T$. It introduces a three-stage pipeline that combines masked self-supervised pre-training on the target domain with collaborative self-training that fuses a video student and a CLIP-based image teacher via a MatchOrConf pseudolabeling scheme on masked inputs. The approach leverages masked video modeling, cross-modal supervision, and pseudolabel refinement to achieve strong transfer on Daily-DA, Sports-DA, and UCF-HMDB_full, outperforming prior UDA methods. By demonstrating that masked modeling and cross-modal teacher-student collaboration can robustly bridge domain gaps in video understanding, UNITE paves a practical path for robust action recognition under distribution shift.

Abstract

In this work, we tackle the problem of unsupervised domain adaptation (UDA) for video action recognition. Our approach, which we call UNITE, uses an image teacher model to adapt a video student model to the target domain. UNITE first employs self-supervised pre-training to promote discriminative feature learning on target domain videos using a teacher-guided masked distillation objective. We then perform self-training on masked target data, using the video student model and image teacher model together to generate improved pseudolabels for unlabeled target videos. Our self-training process successfully leverages the strengths of both models to achieve strong transfer performance across domains. We evaluate our approach on multiple video domain adaptation benchmarks and observe significant improvements upon previously reported results.

Figures (4)

• Figure 1: Overview of the UNITE pipeline for video UDA. A teacher model is used to guide the self-supervised learning process in the first stage, and is again used to improve pseudolabeling of target videos during self-training in the third stage.
• Figure 2: Unmasked Teacher (UMT) training used in Stage 1 of UNITE to perform unsupervised representation learning on target domain videos.
• Figure 3: Overview of the collaborative self-training stage in UNITE. The student and teacher models work together to produce more accurate pseudolabels for target domain videos. The target domain classification loss is enforced on masked target videos to encourage stronger context learning. Source domain classification is included to stabilize training, which is especially important at the beginning of training when pseudolabels may have low accuracy.
• Figure 4: Class-wise accuracy on ARID$\rightarrow$HMDB. Performances are shown for the CLIP teacher model and the student model before and after the collaborative self-training stage (Stage 3). For most classes, the Stage 3 student exceeds the accuracy of both the Stage 2 student and the CLIP teacher.