ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning

TL;DR

ViSIL introduces a unified, information-theoretic metric to quantify information loss when compressing a video into a multimodal summary, using a detailed caption as a textual proxy and defining $\mathcal{I}(C; V \mid \tilde{V}) = \log \frac{P(C \mid V)}{P(C \mid \tilde{V})}$ as the core measure. By approximating probabilities with autoregressive vision-language models and a keyword-based probability estimator, ViSIL enables cross-format evaluation that correlates with both human and VLM-based video understanding tasks, such as VQA. The framework reveals that summary format primarily drives processing load, and that a 3-image summary can nearly match full-video performance while drastically reducing token counts and response times; optimizing ViSIL under a Pareto constraint further improves efficiency and understanding. These findings support ViSIL as a practical proxy for information density in multimodal video summarization and retrieval, with potential to guide both evaluation and generation strategies. The work lays groundwork for extending ViSIL to audio-inclusive summaries and integrating the metric into model training to directly preserve video information in compact representations.

Abstract

Multimodal video captioning condenses dense footage into a structured format of keyframes and natural language. By creating a cohesive multimodal summary, this approach anchors generative AI in rich semantic evidence and serves as a lightweight proxy for high-efficiency retrieval. However, traditional metrics like BLEU or ROUGE fail to quantify information coverage across disparate modalities, such as comparing a paragraph of text to a sequence of keyframes. To address this, we propose the Video Summary Information Loss (ViSIL) score, an information-theoretic framework that quantifies the video information not captured by a summary via vision-language model (VLM) inference. By measuring the information loss, ViSIL is a unified metric that enables direct comparison across multimodal summary formats despite their structural discrepancies. Our results demonstrate that ViSIL scores show a statistically significant correlation with both human and VLM performance on Video Question Answering (VQA) tasks. ViSIL also enables summary selection to optimize the trade-off between information loss and processing speed, establishing a Pareto-optimal frontier that outperforms text summaries by $7\%$ in VQA accuracy without increasing processing load.

Figures (11)

• Figure 1: A Unified Evaluation for Multimodal Video Captions. Given a video $V$, VLM-generated detailed caption $C$, and several multimodal video summaries $\tilde{V}$, the ViSIL score quantifies the information loss within the summaries relative to the original video content. Our results show that ViSIL correlates with video understanding (VQA accuracy) for both humans and VLMs, while the summary format dominates the process load (response time and token count).
• Figure 2: ViSIL Implementation via VLM Inference. ViSIL assesses information loss by comparing a VLM's ability to recover masked tokens in caption $C$ from video $V$ versus summary $\tilde{V}$. ViSIL is defined as the pointwise mutual information between the video and caption conditioned on the summary, representing the information in the video that remains unaccounted for by the summary. A lower ViSIL score indicates better information preservation.
• Figure 3: Pareto Frontier of Process Speed and ViSIL Score showing that static formats are sub-optimal for the process speed–information trade-off. The annotated VQA accuracy confirms ViSIL identifies high-utility summaries that outperform pure text and fixed-image formats while preserving processing speed.
• Figure 4: VQA Accuracy
• Figure 5: Response Time Distribution