Augmenting Sports Videos with VisCommentator

TL;DR

This paper tackles the challenge of embedding data visualizations in sports videos by deriving a design space that separates what data/visuals are used from how they are organized. It analyzes 233 real-world augmented videos to define element-level (Data Type, Visual Type) and clip-level (Data Level, Narrative Order) dimensions, and demonstrates patterns that guide visualization design. Building on this space, VisCommentator provides a data-driven, rapid prototyping tool for table tennis videos, automatically extracting data with ML models, enabling direct interaction with video objects, and recommending visuals via a probabilistic mapping that accounts for narrative order, implemented in a browser/server architecture. A reproduction-based user study with seven domain experts shows high usability and efficiency, indicating strong potential for generalization to other racket sports and for guiding future integration with visual analytics and broader sports contexts.

Abstract

Visualizing data in sports videos is gaining traction in sports analytics, given its ability to communicate insights and explicate player strategies engagingly. However, augmenting sports videos with such data visualizations is challenging, especially for sports analysts, as it requires considerable expertise in video editing. To ease the creation process, we present a design space that characterizes augmented sports videos at an element-level (what the constituents are) and clip-level (how those constituents are organized). We do so by systematically reviewing 233 examples of augmented sports videos collected from TV channels, teams, and leagues. The design space guides selection of data insights and visualizations for various purposes. Informed by the design space and close collaboration with domain experts, we design VisCommentator, a fast prototyping tool, to eases the creation of augmented table tennis videos by leveraging machine learning-based data extractors and design space-based visualization recommendations. With VisCommentator, sports analysts can create an augmented video by selecting the data to visualize instead of manually drawing the graphical marks. Our system can be generalized to other racket sports (e.g., tennis, badminton) once the underlying datasets and models are available. A user study with seven domain experts shows high satisfaction with our system, confirms that the participants can reproduce augmented sports videos in a short period, and provides insightful implications into future improvements and opportunities.

Figures (21)

• Figure 1: The number of videos a) of different ball sports and b) with different time duration. The top-10 frequently presented c) data types and d) visual types in augmented sports videos.
• Figure 2: Video examples in the corpus of a) table tennis in Linear, b) soccer in FlashForward, c) basketball in FlashBack, d) badminton in TimeFork, e) tennis in Grouped, and f) football in ZigZag.
• Figure 3: A clip-level design space for augmented sports videos: Data Level and Narrative Order. The number in cells depict their combination occurrences in our corpus. Darker cells mean more occurrences. The last row and column present the ratio of each option to its dimension.
• Figure 4: The user interface of the system, including a) a video preview, b) a timeline, and c) an edit panel. A basic authoring workflow includes four steps: 1) brush the timeline, 2) choose the key event, 3) select the Narrative Purpose and Narrative Order, and 4) select the augmented data by directly interacting with the video objects. The system will suggest the visuals for the selected data in the edit panel (c1).
• Figure 5: a) The rotation speed of the ball is 7000 rounds per minute. b) The probability distribution of the ball placement. c) The player attacks the ball to win the rally. d) The video uses a FlashForward to preview the action of the player after showing the potential ball placement.