Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization

TL;DR

The paper tackles the challenge of producing detailed and accurate captions for audio-visual video data. It presents video-SALMONN 2, a 7B audio-visual LLM built on a visual LLM with LoRA, enhanced through multi-round direct preference optimization (mrDPO) and a rebirth tuning stage to protect non-captioning abilities. It introduces new metrics for caption completeness and hallucination, demonstrates strong captioning gains through mrDPO with LoRA proxies, and shows competitive video QA performance, validating a compact model's effectiveness in multimodal understanding. The results offer a practical, reproducible approach for high-quality video description and provide a pathway for continuing improvements via iterative mrDPO and rebirth cycles.

Abstract

Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding. In this paper, we present video-SALMONN 2, an advanced audio-visual large language model (LLM) with low-rank adaptation (LoRA) designed for enhanced video (with paired audio) captioning through directed preference optimization (DPO). We propose new metrics to evaluate the completeness and accuracy of video descriptions, which are optimized using DPO. To further improve training, we introduce a novel multi-round DPO (mrDPO) approach, which involves periodically updating the DPO reference model, merging and re-initializing the LoRA module as a proxy for parameter updates after each training round (1,000 steps), and incorporating guidance from ground-truth video captions to stabilize the process. To address potential catastrophic forgetting of non-captioning abilities due to mrDPO, we propose rebirth tuning, which finetunes the pre-DPO LLM by using the captions generated by the mrDPO-trained model as supervised labels. Experiments show that mrDPO significantly enhances video-SALMONN 2's captioning accuracy, reducing global and local error rates by 40\% and 20\%, respectively, while decreasing the repetition rate by 35\%. The final video-SALMONN 2 model, with just 7 billion parameters, surpasses leading models such as GPT-4o and Gemini-1.5-Pro in video captioning tasks, while maintaining competitive performance to the state-of-the-art on widely used video question-answering benchmark among models of similar size. Upon acceptance, we will release the code, model checkpoints, and training and test data. Demos are available at \href{https://video-salmonn-2.github.io}{https://video-salmonn-2.github.io}.

Figures (10)

• Figure 1: The overall structure of video-SALMONN 2. The input video is processed separately by the visual and audio branches, generating visual and audio tokens from the visual and audio frame sequences. Next, the visual and audio tokens are interleaved synchronously, and combined with the tokens of the text prompt to form the input to the LLM backbone.
• Figure 2: An overview of the complete training process includes audio modality alignment, SFT, mrDPO, and rebirth tuning. LoRA is introduced during the SFT stage, and a new LoRA proxy is added to the LLM before each round of DPO. After multiple rounds of DPO, the model generates new data, which is then utilized during rebirth tuning to further refine the model.
• Figure 3: The pipeline for mrDPO in video captioning uses GPT to process the video’s atomic event list to automatically identify preferred and dispreferred caption samples generated by the model. These sample pairs are then utilized in applying DPO-based RL algorithms. This process can be repeated over multiple rounds to iteratively enhance performance.
• Figure 4: The comparison of different methods is shown in Fig. \ref{['fig:loss']}. All models are trained either until convergence or until they begin to frequently generate unnatural patterns. The dashed line indicates that this round of RL resulted in a high frequency of unnatural text generation and, therefore, cannot be considered a valid performance indicator for the model. Fig. \ref{['fig:rl']} demonstrates that RL after rebirth tuning achieves better performance, while mrDPO provides only minimal gains.
• Figure 5: The pipeline for labelling videos with high-quality audio-visual captions.