IF-VidCap: Can Video Caption Models Follow Instructions?

TL;DR

IF-VidCap introduces the first benchmark designed to systematically evaluate instruction-following in video captioning. By jointly assessing format and content fidelity through CSR and ISR, it reveals a nuanced landscape where open-source large multimodal models increasingly rival proprietary systems, especially when endowed with reasoning-enabled prompting. The study also demonstrates a scalable training pipeline that improves instruction-following via targeted fine-tuning on a dedicated dataset. Collectively, IF-VidCap provides a rigorous framework and resources to accelerate the development of controllable, semantically faithful video captions with practical downstream impact.

Abstract

Although Multimodal Large Language Models (MLLMs) have demonstrated proficiency in video captioning, practical applications require captions that follow specific user instructions rather than generating exhaustive, unconstrained descriptions. Current benchmarks, however, primarily assess descriptive comprehensiveness while largely overlooking instruction-following capabilities. To address this gap, we introduce IF-VidCap, a new benchmark for evaluating controllable video captioning, which contains 1,400 high-quality samples. Distinct from existing video captioning or general instruction-following benchmarks, IF-VidCap incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness. Our comprehensive evaluation of over 20 prominent models reveals a nuanced landscape: despite the continued dominance of proprietary models, the performance gap is closing, with top-tier open-source solutions now achieving near-parity. Furthermore, we find that models specialized for dense captioning underperform general-purpose MLLMs on complex instructions, indicating that future work should simultaneously advance both descriptive richness and instruction-following fidelity.

Figures (8)

• Figure 1: Results of different MLLMs on Controlled Video Captioning. Constraint types are color-coded according to the categories in Figure \ref{['fig:constraint']}. Sentences highlighted in red are incorrect event captions.
• Figure 2: Sample data in IF-VidCap. The constraint types are color-coded, with colors corresponding to the categories in Figure \ref{['fig:constraint']}. Our checklist is divided into two types based on the checking method: rule-based items checked by LLM with rule scripts and open-ended items checked by LLM. The rule-based items cover format correctness, while the open-ended items cover semantic and content correctness.
• Figure 3: Dataset statistics for IF-VidCap. (a-c) show distributions for video duration, category, and constraint count, respectively. (d) provides an overview of the constraint categories.
• Figure 4: The Impact of Constraint Count, Instruction Length, Frame Count, and Resolutions on Instruction-Following Capability (CSR and ISR).
• Figure 5: CSR performance of different models on different constraint types.