SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioning

Abstract

Fine-grained video captioning aims to generate detailed, temporally coherent descriptions of video content. However, existing methods struggle to capture subtle video dynamics and rich detailed information. In this paper, we leverage preference learning to enhance the performance of vision-language models in fine-grained video captioning, while mitigating several limitations inherent to direct preference optimization (DPO). First, we propose a pipeline for constructing preference pairs that leverages the intrinsic properties of VLMs along with partial assistance from large language models, achieving an optimal balance between cost and data quality. Second, we propose Synergistic Preference Optimization (SynPO), a novel optimization method offering significant advantages over DPO and its variants. SynPO prevents negative preferences from dominating the optimization, explicitly preserves the model's language capability to avoid deviation of the optimization objective, and improves training efficiency by eliminating the need for the reference model. We extensively evaluate SynPO not only on video captioning benchmarks (e.g., VDC, VDD, VATEX) but also across well-established NLP tasks, including general language understanding and preference evaluation, using diverse pretrained models. Results demonstrate that SynPO consistently outperforms DPO variants while achieving 20\% improvement in training efficiency. Code is available at https://github.com/longmalongma/SynPO

Figures (6)

• Figure 1: Left: SynPO significantly outperforms other methods in different models on VDC benchmark chai2024auroracap. Middle: Language capability degradation occurs during the latter training stages in DPO. Training collapses and is biased towards maximazing positive-negative reward gap. Right: SynPO mitigates degradtion successfully and resolves the issue of optimization objectives shifting from language capability to ranking differentiation. Its performance significantly outperforms that of DPO.
• Figure 2: Overview of dataset construction pipeline. A VLM first generates multiple candidate captions for each video with the self-retrospective strategy. Then the candidate captions are scored by an LLM based on three criteria (i.e., factuality, linguistic fluency, and self-consistency) to select positive and negative preferences.
• Figure 3: The evolution of positive and negative rewards and the normalized Frobenius norm of the gradient with respect to positive and negative rewards during training of DPO and SynPO. DPO training undergoes simultaneous decreases in both rewards, with negative preferences dominating the optimization process. Conversely, SynPO mitigates this problem, demonstrating improved performance and stability.
• Figure 4: Language capability of different fine-tuning methods.
• Figure 5: Log-probability vs. token importance.