VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection

TL;DR

This work introduces VDC-Agent, a self-evolving framework that turns a multimodal language model into a stronger video detailed captioner without human-annotated data or larger teacher models. It iterates caption generation, principle-guided evaluation, and prompt refinement, with a self-reflection mechanism that revisits prior reasoning when updates regress. Trajectories of caption–score pairs are harvested from unlabeled videos and converted into 18,886 preference tuples (VDC-Agent-19K), which are used to fine-tune a base model via curriculum Direct Preference Optimization. On the VDC benchmark, the 7B variant built on Qwen2.5-VL-7B-Instruct achieves state-of-the-art results (49.08% average accuracy and 2.50 score), demonstrating that agentic self-reflection can surpass specialized captioners at similar inference costs while improving scalability and reproducibility.

Abstract

We present VDC-Agent, a self-evolving framework for Video Detailed Captioning that requires neither human annotations nor larger teacher models. The agent forms a closed loop of caption generation, principle-guided scoring (score and textual suggestions), and prompt refinement. When caption quality regresses, a self-reflection path leverages the previous chain-of-thought to amend the update. Running this process on unlabeled videos produces trajectories of (caption, score) pairs. We convert the trajectories into preference tuples and filter out samples with JSON parsing errors, resulting in VDC-Agent-19K, which contains 18,886 automatically constructed pairs. We then fine-tune the base MLLM on this dataset using an easy-to-hard curriculum direct preference optimization. Built on Qwen2.5-VL-7B-Instruct, our VDC-Agent-7B attains state-of-the-art performance on the VDC benchmark with 49.08% average accuracy and 2.50 score, surpassing specialized video captioners and improving over the base model by +5.13% accuracy and +0.27 score at similar inference cost.

Figures (3)

• Figure 1: Comparison of video captioning paradigms. (a) Human-aligned pipelines rely on manual annotations to train caption scorers. (b) Multi-MLLM-based pipelines depend on multiple or stronger MLLMs for scoring or fusion. (c) Our proposed VDC-Agent achieves self-improvement through agentic self-reflection, requiring neither human annotations nor larger models.
• Figure 2: Overview of VDC-Agent and Dataset Construction. Given a video $x$ and an initial prompt $p_0$, the base MLLM produces a caption $y_t$. A principle-guided MLLM then returns a quality score $s_t$ and textual suggestions $g_t$. If $s_t \ge \lambda$, the loop stops; otherwise the prompt is revised by the Prompt Refiner, and when $s_t < s_{t-1}$ a Self-Reflection path is triggered using the previous chain-of-thought to diagnose and amend the update. This process repeats up to $T$ iterations, yielding a set of $(\text{caption}, \text{score})$ pairs. We sort these pairs to obtain the best and worst captions $(y^{+}, y^{-})$ and compute the preference strength $\Delta\mathrm{Score}=s^{+}-s^{-}$. Each video thus forms a training tuple $(x, y^{+}, y^{-}, \Delta s)$, which is used to fine-tune the MLLM with Curriculum DPO via an easy-to-hard sampler (large to small $\Delta s$).
• Figure 3: Qualitative comparison on a video from the VDC benchmark. We compare captions generated by Qwen2.5-VL-7B-Instruct, Video-R1-7B, Cockatiel-13B, and our VDC-Agent-7B under the same prompt: "Please provide a detailed description of the given video." Text in green denotes correct, video-grounded descriptions, while red indicates incorrect or hallucinated content. Blue highlights fine-grained details that are correctly identified only by our VDC-Agent-7B.