Semi-Supervised Training to Improve Player and Ball Detection in Soccer

TL;DR

This work tackles the challenge of detecting players and the ball in soccer videos when labeled data are scarce. It introduces a generic semi-supervised framework based on a teacher–student paradigm: a teacher trained on labeled data generates pseudo-labels for a large unlabeled set, which a student then learns from alongside labeled examples using three confidence-based loss parametrizations that allow doubt about teacher predictions. The approach yields substantial performance gains over fully supervised training, achieving a first SoccerNet-v3 detection benchmark with mAP up to 52.3% and demonstrating that more unlabeled data generally yields larger improvements. The method is architecture- and domain-agnostic, making it applicable to other sports and detection tasks with limited annotations.

Abstract

Accurate player and ball detection has become increasingly important in recent years for sport analytics. As most state-of-the-art methods rely on training deep learning networks in a supervised fashion, they require huge amounts of annotated data, which are rarely available. In this paper, we present a novel generic semi-supervised method to train a network based on a labeled image dataset by leveraging a large unlabeled dataset of soccer broadcast videos. More precisely, we design a teacher-student approach in which the teacher produces surrogate annotations on the unlabeled data to be used later for training a student which has the same architecture as the teacher. Furthermore, we introduce three training loss parametrizations that allow the student to doubt the predictions of the teacher during training depending on the proposal confidence score. We show that including unlabeled data in the training process allows to substantially improve the performances of the detection network trained only on the labeled data. Finally, we provide a thorough performance study including different proportions of labeled and unlabeled data, and establish the first benchmark on the new SoccerNet-v3 detection task, with an mAP of 52.3%. Our code is available at https://github.com/rvandeghen/SST .

Figures (5)

• Figure 1: Overview. Given a small labeled image dataset for object detection in soccer such as the players, the ball, or the referees, we leverage a large unlabeled dataset of soccer broadcast videos for training an object detector in a semi-supervised fashion. Our training technique allows us to significantly improve the performance of the object detector for the targeted soccer application.
• Figure 2: Overview of our semi-supervised training method for player and ball detection. We first train a teacher network on a labeled dataset in a fully supervised fashion. Then, we use the trained teacher to produce pseudo-labels on the unlabeled dataset. This creates a first pseudo-labeled dataset, with a confidence score for each prediction. The labeled and pseudo-labeled datasets are then used to train a student network, whose training loss is parameterized based on the confidence score with one of the three parametrization introduced in this paper. This allows the student to doubt unsure proposals by the teacher and achieve good performances on the test dataset. At the end of the training, a final fine-tuning phase is performed with the labeled data, and the student becomes the new teacher for the next iteration.
• Figure 3: Our three loss parametrizations for positive candidates. Comparison of the evolution of the proposal loss weight (corresponding to $\alpha_j$) with respect to the prediction confidence score for our three parametrizations for positive candidates (in red). (1) Simple threshold value to discriminate between the positive proposals and the background by assigning the same loss weight to all positive samples. (2) Introduction of a second threshold to delimit a doubt zone where the loss is zeroed out. (3) Soft linear approximation for the loss weight in the doubt zone to give more importance to predictions close to $\tau_h$. Note that the loss weight is always $1$ for background proposals (in blue), regardless of the parametrization for the positive proposals.
• Figure 4: Optimal threshold value for the first parametrization. Comparison of the performance of the first parametrization for different threshold values $\tau_h$ on various labeled dataset sizes, with $10$ extra unlabeled games. The performance of the student increases with the threshold value indicating that only predictions for which the teacher is certain should be considered. Also, the student manages to surpass the teacher for each dataset size.
• Figure 5: Qualitative results. Comparison of the detections on a test set image for the first teacher (left), fine-tuned student model after $1$ iteration (middle), and fine-tuned student model after $2$ iterations (right). The considered labeled dataset size is $10\%$, with $10$ extra unlabeled games, using the third parametrization for both iterations, with the optimal threshold values presented in Table \ref{['tab:main-results']}.