video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model

TL;DR

This work introduces video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM for general video understanding, built by coupling a visual encoder, an audio encoder, and a large language model with reasoning-focused training. It advances reasoning through a specialized SFT dataset and a novel training objective, process direct preference optimization (pDPO), which uses per-step contrastive preferences to guide step-level reasoning. A new benchmark, RivaBench, provides over 4k expert-curated QA pairs across academic, stand-up, and synthetic video scenarios to rigorously evaluate audio-visual reasoning; results show 3-8% absolute gains over a strong baseline and 6-8% gains for pDPO on RivaBench, plus zero-shot synthetic video detection. The approach enhances interpretability and enables more reliable AV reasoning, with implications for safer, more controllable AI systems in multimedia contexts.

Abstract

While recent advancements in reasoning optimization have significantly enhanced the capabilities of large language models (LLMs), existing efforts to improve reasoning have been limited to solving mathematical problems and focusing on visual graphical inputs, neglecting broader applications in general video understanding.This paper proposes video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM designed for general video understanding tasks. To enhance its reasoning abilities, we develop a reasoning-intensive dataset featuring challenging audio-visual questions with step-by-step solutions. We also propose process direct preference optimization (pDPO), which leverages contrastive step selection to achieve efficient step-level reward modelling tailored for multimodal inputs. Additionally, we introduce RivaBench, the first reasoning-intensive video understanding benchmark, featuring over 4,000 high-quality, expert-curated question-answer pairs across scenarios such as standup comedy, academic presentations, and synthetic video detection. video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision baseline across different video reasoning benchmarks. Besides, pDPO achieves 6-8% improvements compared to the supervised fine-tuning model on RivaBench. Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection capabilities.

Figures (18)

• Figure 1: video-SALMONN-o1 model structure. The input video is processed by the visual and audio branches, generating encodings from the visual and audio frame sequences respectively. Two encoding streams are combined in an interleaved fashion to synchronize across time before sending to LLM.
• Figure 2: Acquisition pipeline of reasoning-intensive SFT data. The question, answer and reasoning paths are generated by Gemini-1.5-pro taking the video with paired audio as inputs. GPT4o is employed for quality checks to ensure the QA-pair and the reasoning steps are valid and require logical thinking.
• Figure 3: Illustration of the contrastive step selection (top) and pairwise rollout (bottom) to construct per-step expected correctness score for pDPO. Contrastive step selection: Top 2 steps, $s_2$ and $s_5$ are selected in this example, and for $s_2$, an alternative step, $s'_2$, is sampled to form the preference pair. Pairwise rollout: Three rollouts are shown for each step and $s_2$ and $s'_2$ are step pairs with the same prefix solution. The answer correctness is checked using GPT-4o by comparing it against the reference answer.
• Figure 4: Distributions of the numbers of reasoning steps in SFT data. Left: Distribution of the entire SFT data. Right: Distribution on the reasoning-intensive subset of SFT data. Due to the difficulty of the reasoning-intensive subset, more reasoning steps are required in general for samples in this set.
• Figure 5: Comparison between different top T steps selected for pDPO. Pairs of full solution paths are always used in addition to pairs of intermediate steps.