Gameplay Highlights Generation

TL;DR

This work demonstrates that a multimodal video-language model, X-CLIP, can be finetuned to detect FPS game events from unseen footage with high accuracy, enabling automatic highlight reel generation. By pairing a video encoder with a text encoder and enriching prompts with per-game context, the approach achieves strong cross-game generalization without game-specific engineering. The authors show that INT8 post-training quantization makes deployment feasible on consumer hardware, while maintaining competitive accuracy, and they validate robustness through unseen-game testing and runtime analyses. The practical impact lies in scalable, production-ready highlight generation for diverse games, with potential extensions into audio signals and additional genres. The study highlights the value of natural language supervision for data-efficient video understanding in gaming contexts.

Abstract

In this work, we enable gamers to share their gaming experience on social media by automatically generating eye-catching highlight reels from their gameplay session Our automation will save time for gamers while increasing audience engagement. We approach the highlight generation problem by first identifying intervals in the video where interesting events occur and then concatenate them. We developed an in-house gameplay event detection dataset containing interesting events annotated by humans using VIA video annotator. Traditional techniques for highlight detection such as game engine integration requires expensive collaboration with game developers. OCR techniques which detect patches of specific images or texts require expensive per game engineering and may not generalize across game UI and different language. We finetuned a multimodal general purpose video understanding model such as X-CLIP using our dataset which generalizes across multiple games in a genre without per game engineering. Prompt engineering was performed to improve the classification performance of this multimodal model. Our evaluation showed that such a finetuned model can detect interesting events in first person shooting games from unseen gameplay footage with more than 90% accuracy. Moreover, our model performed significantly better on low resource games (small dataset) when trained along with high resource games, showing signs of transfer learning. To make the model production ready, we used ONNX libraries to enable cross platform inference. These libraries also provide post training quantization tools to reduce model size and inference time for deployment. ONNX runtime libraries with DirectML backend were used to perform efficient inference on Windows OS. We show that natural language supervision in the X-CLIP model leads to data efficient and highly performant video recognition models.

Figures (2)

• Figure 1: Pipeline block diagram representing the inference process
• Figure 2: Pipeline block diagram representing the finetuning process