Shot2Tactic-Caption: Multi-Scale Captioning of Badminton Videos for Tactical Understanding

TL;DR

This work tackles the challenge of translating dynamic badminton tactics into natural language by proposing Shot2Tactic-Caption, a dual-branch, Transformer-based framework capable of generating both shot-level and tactic-level captions. It introduces the Shot2Tactic-Caption Dataset, featuring aligned shot and tactic annotations with defined tactic types and states to capture progression, interruptions, and resumptions of tactics. A Tactic Unit Detector guides tactic-caption generation, and a shot-wise prompt-guided mechanism injected via cross-attention improves temporal coherence and applicability to real-time commentary. Experimental results and ablations show that preserving fine-grained spatial details (via a ResNet50-STE encoder) and maintaining temporal alignment through shot-wise prompts substantially improve caption quality and coherence, enabling richer tactical understanding and potential coaching or analysis tools.

Abstract

Tactical understanding in badminton involves interpreting not only individual actions but also how tactics are dynamically executed over time. In this paper, we propose \textbf{Shot2Tactic-Caption}, a novel framework for semantic and temporal multi-scale video captioning in badminton, capable of generating shot-level captions that describe individual actions and tactic-level captions that capture how these actions unfold over time within a tactical execution. We also introduce the Shot2Tactic-Caption Dataset, the first badminton captioning dataset containing 5,494 shot captions and 544 tactic captions. Shot2Tactic-Caption adopts a dual-branch design, with both branches including a visual encoder, a spatio-temporal Transformer encoder, and a Transformer-based decoder to generate shot and tactic captions. To support tactic captioning, we additionally introduce a Tactic Unit Detector that identifies valid tactic units, tactic types, and tactic states (e.g., Interrupt, Resume). For tactic captioning, we further incorporate a shot-wise prompt-guided mechanism, where the predicted tactic type and state are embedded as prompts and injected into the decoder via cross-attention. The shot-wise prompt-guided mechanism enables our system not only to describe successfully executed tactics but also to capture tactical executions that are temporarily interrupted and later resumed. Experimental results demonstrate the effectiveness of our framework in generating both shot and tactic captions. Ablation studies show that the ResNet50-based spatio-temporal encoder outperforms other variants, and that shot-wise prompt structuring leads to more coherent and accurate tactic captioning.

Figures (3)

• Figure 1: Statistical analysis of captions: (a) Distribution of caption lengths in shot and tactic captions. (b) Frequency distribution of the most common words in shot and tactic captions.
• Figure 2: Overview of the Shot2Tactic-Caption framework. T-DEEDxarles2024t segments the input video into shots, which are then grouped into tactic units with predicted tactic type and state by the Tactic Unit Detector. Two parallel encoder-decoder branches generate shot captions (orange) and tactic captions (blue). The tactic branch leverages shot-wise prompts via cross-attention to guide tactic caption generation.
• Figure 3: Qualitative examples of shot captions (A--G) and tactic captions (①--③). Different prompt types are used in tactic captioning to illustrate their effect on description quality.