Towards Precise Action Spotting: Addressing Temporal Misalignment in Labels with Dynamic Label Assignment

TL;DR

This work proposes a novel dynamic label assignment strategy that allows predictions to have temporal offsets from ground-truth action times during training, ensuring consistent event spotting and achieves state-of-the-art performance.

Abstract

Precise action spotting has attracted considerable attention due to its promising applications. While existing methods achieve substantial performance by employing well-designed model architecture, they overlook a significant challenge: the temporal misalignment inherent in ground-truth labels. This misalignment arises when frames labeled as containing events do not align accurately with the actual event times, often as a result of human annotation errors or the inherent difficulties in precisely identifying event boundaries across neighboring frames. To tackle this issue, we propose a novel dynamic label assignment strategy that allows predictions to have temporal offsets from ground-truth action times during training, ensuring consistent event spotting. Our method extends the concept of minimum-cost matching, which is utilized in the spatial domain for object detection, to the temporal domain. By calculating matching costs based on predicted action class scores and temporal offsets, our method dynamically assigns labels to the most likely predictions, even when the predicted times of these predictions deviate from ground-truth times, alleviating the negative effects of temporal misalignment in labels. We conduct extensive experiments and demonstrate that our method achieves state-of-the-art performance, particularly in conditions where events are visually distinct and temporal misalignment in labels is common.

Figures (9)

• Figure 1: Two examples of our proposed dynamic label assignment. In both cases, the action label "Near court serve" is tagged to the frames at time $t$. In example (a), the event appears to occur around $t$, leading our method to assign the label to the prediction with a predicted time close to $t$ during training. Conversely, in example (b), the event appears to take place around $t - 1$. In this case, our method assigns the label to the prediction with a predicted time close to $t - 1$ based on the predicted action class scores.
• Figure 2: Model architecture of the proposed method.
• Figure 3: Example cases from the Tennis dataset, where the predicted times exhibit offsets from the ground-truth times during training. The center images display cropped action regions from the tagged frames, while the left and right images show the regions from the frames immediately before and after the tagged frames. The values indicated in the blue boxes and arrows represent the offsets in the predictions.
• Figure 4: Comparison against state-of-the-art methods. The solid and dotted lines indicate the results for $\delta = 1$ and $\delta = 2$, respectively. For reference, we include the results from Table \ref{['table:comp_precise']} at $\sigma$ of 0.0.
• Figure 5: Offsets between ground-truth times in the original precise labels and predicted times. Since our model is trained using the noisy labels associated with $\sigma$ of 2.0, we can expect an average offset of 1.58 between the predicted times and the ground truth times in the precise labels. However, the predicted times eventually exhibit offsets of less than 1.58 during training.