LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning

TL;DR

The paper tackles Generic Event Boundary Captioning (GEBC) by integrating a pretrained large language model with a trainable video adapter (video Q-former) and multiple visual feature streams. It freezes the visual feature extractors and LLM, training only the video adapter to map video inputs into LLM-compatible tokens, which are then used to generate three boundary-specific captions per event. The approach achieves state-of-the-art results on Kinetic-GEBC (76.14 AVG, with strong SPICE, ROUGE-L, and CIDEr scores) and demonstrates that fusing BLIP-2, CLIP, Omnivore, VinVL features alongside a video adapter significantly improves boundary-aware captioning. The work provides strong evidence that video adapters can effectively adapt LLMs to specialized video-language tasks and publishes code for reproducibility.

Abstract

Our winning entry for the CVPR 2023 Generic Event Boundary Captioning (GEBC) competition is detailed in this paper. Unlike conventional video captioning tasks, GEBC demands that the captioning model possess an understanding of immediate changes in status around the designated video boundary, making it a difficult task. This paper proposes an effective model LLMVA-GEBC (Large Language Model with Video Adapter for Generic Event Boundary Captioning): (1) We utilize a pretrained LLM for generating human-like captions with high quality. (2) To adapt the model to the GEBC task, we take the video Q-former as an adapter and train it with the frozen visual feature extractors and LLM. Our proposed method achieved a 76.14 score on the test set and won the first place in the challenge. Our code is available at https://github.com/zjr2000/LLMVA-GEBC .

Figures (2)

• Figure 1: Illustration of the proposed LLMVA-GEBC model.
• Figure 2: Qualitative Results on Kinetic-GEBC validation set. Our model produces captions of high quality.