Human-in-the-loop Adaptation in Group Activity Feature Learning for Team Sports Video Retrieval

TL;DR

This work tackles the challenge of retrieving team-sports videos when group-activity annotations are unavailable. It introduces a two-stage approach: first, self-supervised pre-training of Group Activity Feature Learning (GAFL) to obtain a discriminative GAF space, then a human-in-the-loop fine-tuning stage that uses binary positive/negative labels on a small set of query-relevant videos to tailor the GAF space to a target activity. The fine-tuning relies on a data-efficient query-aware video selection to assemble informative training examples, coupled with diversity-based Core-set selection and a contrastive learning objective plus regularization to preserve the pre-trained space. Experiments on Volleyball, NBA, and Collective Activity datasets show consistent retrieval improvements and ablations demonstrate the effectiveness of query similarity, local dissimilarity, and diversity sampling. The method significantly reduces annotation burden while enabling targeted, practical analysis of tactics in team sports.

Abstract

This paper proposes human-in-the-loop adaptation for Group Activity Feature Learning (GAFL) without group activity annotations. This human-in-the-loop adaptation is employed in a group-activity video retrieval framework to improve its retrieval performance. Our method initially pre-trains the GAF space based on the similarity of group activities in a self-supervised manner, unlike prior work that classifies videos into pre-defined group activity classes in a supervised learning manner. Our interactive fine-tuning process updates the GAF space to allow a user to better retrieve videos similar to query videos given by the user. In this fine-tuning, our proposed data-efficient video selection process provides several videos, which are selected from a video database, to the user in order to manually label these videos as positive or negative. These labeled videos are used to update (i.e., fine-tune) the GAF space, so that the positive and negative videos move closer to and farther away from the query videos through contrastive learning. Our comprehensive experimental results on two team sports datasets validate that our method significantly improves the retrieval performance. Ablation studies also demonstrate that several components in our human-in-the-loop adaptation contribute to the improvement of the retrieval performance. Code: https://github.com/chihina/GAFL-FINE-CVIU.

Figures (12)

• Figure 1: Difference between previous method and our method. (a) Supervised GAR employs group activity annotations that are difficult due to various similar group activities. (b) Our proposed method adaptively fine-tunes the pre-trained GAF space for retrieving target group activity represented by query videos given by users. Our fine-tuning only requires binary annotations to users on a few selected videos, eliminating group activity annotations.
• Figure 2: Group activity feature space learned by DBLP:conf/cvpr/NakataniKU24 in a self-supervised manner on the Volleyball dataset DBLP:conf/cvpr/IbrahimMDVM16. The group activity feature extracted from each video is transformed into 2-dimensional features by t-SNE van2008visualizing. The color of each point corresponds to its group activity label annoated in the Volleyball dataset. While the manually annotated group activity labels are not used for the training, the group activity labels are only used for the visualization in this figure. The visualization reveals that group activity features are not learned enough in DBLP:conf/cvpr/NakataniKU24 to discriminate between the different group activities.
• Figure 3: Overview of our proposal. (a) group activity feature learning network is pre-trained by DBLP:conf/cvpr/NakataniKU24 with the training dataset. (b) The pre-trained group activity feature learning network is fine-tuned for target group activity presented by query videos given by users.
• Figure 4: Overview of our group activity feature learning network. (a) Person feature extractor. The person feature is composed of appearance and location features. (b) Group Activity Feature Learning (GAFL) network. The GAF is learned from extracted people features. (c) Location-guided appearance feature prediction network with the GAF. The appearance feature of each person is predicted from the location feature of the person and the GAF extracted in (b). Through the appearance feature prediction, the GAF is learned in a self-supervised manner.
• Figure 5: Query local dissimilarity computation. The query GAF and Locally Masked query GAF are extracted by the pre-trained GAFL network. For each video in $\bm{D}^{\mathrm{train}}$, the variance of the GAF similarities between the GAFs extracted from the query videos during the masking is computed.