PolySmart @ TRECVid 2024 Video Captioning (VTT)

TL;DR

This paper investigates the capabilities of Vision-Language Models like LLaVA and LLaVA-NeXT-Video in generating natural language descriptions for video content and investigates the impact of fine-tuning VLMs on VTT datasets to enhance description accuracy, contextual relevance, and linguistic consistency.

Abstract

In this paper, we present our methods and results for the Video-To-Text (VTT) task at TRECVid 2024, exploring the capabilities of Vision-Language Models (VLMs) like LLaVA and LLaVA-NeXT-Video in generating natural language descriptions for video content. We investigate the impact of fine-tuning VLMs on VTT datasets to enhance description accuracy, contextual relevance, and linguistic consistency. Our analysis reveals that fine-tuning substantially improves the model's ability to produce more detailed and domain-aligned text, bridging the gap between generic VLM tasks and the specialized needs of VTT. Experimental results demonstrate that our fine-tuned model outperforms baseline VLMs across various evaluation metrics, underscoring the importance of domain-specific tuning for complex VTT tasks.

Figures (3)

• Figure 1: Vision Language Model Framework.
• Figure 2: Case study among Vanilla LLaVA (LV) and Fine-tuned LLaVA (LV-FT)
• Figure 3: Comparison among Text Embedding t-SNE Distributions.