Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark

TL;DR

This work tackles the challenge of generating video captions that include specific player names and fine-grained actions in basketball broadcasts. It constructs a multimodal knowledge graph (KG_NBA_2022) and a corresponding video-captioning dataset (VC_NBA_2022), then proposes KEANet, an encoder-decoder model with temporal Bi-GRU and an entity-aware module that leverages candidate player knowledge (images and names) to produce knowledge-grounded captions via a T5 decoder. The approach yields superior performance on VC_NBA_2022 and demonstrates transferability to soccer with the Goal dataset, underscoring the value of integrating multimodal knowledge and entity relationships for practical live narration. The work offers a concrete path toward realistic, entity-specific broadcast captions and provides resources that can be extended to other domains and sports.

Abstract

Despite the recent emergence of video captioning models, how to generate the text description with specific entity names and fine-grained actions is far from being solved, which however has great applications such as basketball live text broadcast. In this paper, a new multimodal knowledge graph supported basketball benchmark for video captioning is proposed. Specifically, we construct a multimodal basketball game knowledge graph (KG_NBA_2022) to provide additional knowledge beyond videos. Then, a multimodal basketball game video captioning (VC_NBA_2022) dataset that contains 9 types of fine-grained shooting events and 286 players' knowledge (i.e., images and names) is constructed based on KG_NBA_2022. We develop a knowledge guided entity-aware video captioning network (KEANet) based on a candidate player list in encoder-decoder form for basketball live text broadcast. The temporal contextual information in video is encoded by introducing the bi-directional GRU (Bi-GRU) module. And the entity-aware module is designed to model the relationships among the players and highlight the key players. Extensive experiments on multiple sports benchmarks demonstrate that KEANet effectively leverages extera knowledge and outperforms advanced video captioning models. The proposed dataset and corresponding codes will be publicly available soon

Figures (12)

• Figure 1: Comparison of conventional captioning with knowledge-grounded captioning. The different specific entity names are marked red and blue, respectively. And the fine-grained actions are marked green.
• Figure 2: An example of a Multimodal Basketball Game Knowledge Graph.
• Figure 3: Data sample from the proposed dataset. Each video is annotated by fileid, action type, caption, player images and player names. Each of the players involved in caption as well as their teammates serve as candidate players.
• Figure 4: Illustration of extracting relevant data using relationship extraction from the knowledge graph and constructing the dataset.
• Figure 5: Word cloud of VC_NBA_2022 and Goal datasets. The bigger the font, the more percentage it occupies.