Wolf: Dense Video Captioning with a World Summarization Framework

TL;DR

Wolf Introduces a world-summarization approach to dense video captioning by combining image- and video-level Vision-Language Models in a cascade, followed by LLM-based fusion to produce detailed, temporally aware captions. The framework is complemented by CapScore, an LLM-based metric aligned with human judgments, and a new Wolf benchmark with four human-annotated datasets (autonomous driving, robotics, Pexels general scenes). Empirical results show Wolf outperforms state-of-the-art and commercial captioners, with substantial CapScore gains on challenging driving videos, and the ability to finetune models with Wolf-generated captions to achieve further improvements. The authors also discuss efficiency optimizations (batched inference, 4-bit quantization) and safety considerations for deploying captioning in embodied AI, along with a public leaderboard to accelerate community progress.

Abstract

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs). By utilizing both image and video models, our framework captures different levels of information and summarizes them efficiently. Our approach can be applied to enhance video understanding, auto-labeling, and captioning. To evaluate caption quality, we introduce CapScore, an LLM-based metric to assess the similarity and quality of generated captions compared to the ground truth captions. We further build four human-annotated datasets in three domains: autonomous driving, general scenes, and robotics, to facilitate comprehensive comparisons. We show that Wolf achieves superior captioning performance compared to state-of-the-art approaches from the research community (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment. Webpage: https://wolfv0.github.io/.

Figures (7)

• Figure 1: Overview of proposed Wolf framework. Wolf utilizes both image-level and video-level models to generate diverse and detailed captions, which are then summarized for cross-checking. On the right side, we also provide an example of how we obtain motion captions based on object locations extracted from image captions.
• Figure 2: Wolf Dataset examples. We display the videos and corresponding human-annotated captions of autonomous driving (Left), Pexels (Top-Right), and Robot learning video dataset (Bottom-Right), totaling 25.7 hours. Our Wolf dataset is fully manually annotated to ensure a robust evaluation for the community. We present our dataset's statistics in Table \ref{['tab:data_statistics']}. We will keep updating and expanding the dataset.
• Figure 3: Illustration of homotopy types of different relative motions between a pair of vehicles.
• Figure 4: Comparisons on Human-Evaluation Score and Llama 3.2-based CapScore and GPT4-based CapScore (proposed).
• Figure 5: Wolf example for driving that focus on interactive operations. Wolf captions discusses the motion behavior in details and serves as a good reference for autonomous driving. Note: Please refer to the Appendix for our caption comparison with other state-of-the-art methods.