Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding

TL;DR

The paper tackles the problem of insufficient temporal understanding in large vision-language models (LVLMs) for video tasks. It conducts an empirical study to identify the visual-language interface as a pivotal factor and then proposes a temporal-oriented recipe that sequentially enhances the interface, introduces temporal training schemes, adds a memory bank for long-range context, and leverages mixture-of-experts to scale the interface. Key contributions include demonstrating that a 12-layer pretrained Q-Former with self-attention markedly improves temporal modeling, validating VC, MC, MG, and DC training schemes (especially when aggregated), and showing that memory banks and MoE further boost performance, particularly for larger LVLMs. The resulting temporally-aware LVLMs achieve state-of-the-art or near-state-of-the-art results on VideoQA and video captioning across multiple datasets, with substantial gains that escalate with model size, offering a practical blueprint for building temporal capacity in LVLMs.

Abstract

Recent years have witnessed outstanding advances of large vision-language models (LVLMs). In order to tackle video understanding, most of them depend upon their implicit temporal understanding capacity. As such, they have not deciphered important components that contribute to temporal understanding ability, which might limit the potential of these LVLMs for video understanding. In this work, we conduct a thorough empirical study to demystify crucial components that influence the temporal understanding of LVLMs. Our empirical study reveals that significant impacts are centered around the intermediate interface between the visual encoder and the large language model. Building on these insights, we propose a temporal-oriented recipe that encompasses temporal-oriented training schemes and an upscaled interface. Our final model developed using our recipe significantly enhances previous LVLMs on standard video understanding tasks.

Figures (1)

• Figure 1: Our temporal-oriented recipe for large vision-language model.