Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling

TL;DR

Video-Teller presents an efficient video-language foundation model that fuses multi-frame visual data with ASR using a cascaded Q-Former, while employing a text auto-encoder to decouple textual representations and enable fine-grained cross-modal alignment. By freezing pretrained vision and language modules and introducing a training-time text-encoder target, it reduces inference costs and mitigates hallucination, achieving competitive or superior results in video captioning and long-form video summarization with less pretraining data. Key contributions include the cascaded Q-Former fusion, the text auto-encoder for decoupled targets, and a fine-grained alignment objective that improves generation quality with minimal parameter overhead. The approach demonstrates strong practical impact for scalable, accurate video-to-text systems with improved reliability in real-world multimodal analysis scenarios.

Abstract

This paper proposes Video-Teller, a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment to significantly enhance the video-to-text generation task. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions. To effectively integrate visual and auditory information, Video-Teller builds upon the image-based BLIP-2 model and introduces a cascaded Q-Former which fuses information across frames and ASR texts. To better guide video summarization, we introduce a fine-grained modality alignment objective, where the cascaded Q-Former's output embedding is trained to align with the caption/summary embedding created by a pretrained text auto-encoder. Experimental results demonstrate the efficacy of our proposed video-language foundation model in accurately comprehending videos and generating coherent and precise language descriptions. It is worth noting that the fine-grained alignment enhances the model's capabilities (4% improvement of CIDEr score on MSR-VTT) with only 13% extra parameters in training and zero additional cost in inference.

Figures (4)

• Figure 1: Overview of the proposed method. Here we show the detailed description generation (long-form text).
• Figure 2: Overall architecture of the proposed model. The model consists of two primary branches. On the right-hand side, we have the text auto-encoder responsible for encoding the target text into a fixed-length representation denoted as $\mathbf{R}_{text}$. Conversely, on the left-hand side, we have the video module, which encodes the input video (comprising frames and ASR) into a video representation that shares the same shape as the text representation. Both of these representations are trained to reconstruct the target text by utilizing the LLM, while they are directly aligned through the Mean Squared Error (MSE) loss.
• Figure 3: An example from Video-CSR. Here frames represent the video's vision information.
• Figure 4: A case shows improvement of hallucination using fine-grained modality alignment.