Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning

Changli Tang, Tianyi Wang, Fengyun Rao, Jing Lyu, Chao Zhang

TL;DR

D-ORCA tackles dialogue-centric audio-visual captioning by introducing a specialized RL framework (GRPO) with three rewards that target speaker attribution, content fidelity, and temporal grounding. It pairs a Q-Former–assisted audio-visual backbone with LoRA-tuned LLMs, trained through SFT, pre-DPO, and GRPO on a large bilingual dataset (DVD-Train) and evaluated on a dedicated benchmark (DVD-Bench). The approach yields state-of-the-art performance among open-source models in speaker identification, ASR accuracy, and temporal localization, while maintaining competitive results on general AV QA benchmarks, despite using 8B parameters. The work also provides robust, LLM-based evaluation protocols and demonstrates the importance of data curation (dialogue-rich English/Chinese videos) for improving dialogue-aware perception in multimodal systems.

Abstract

Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a \textbf{d}ialogue-centric \textbf{o}mni-modal large language model optimized for \textbf{r}obust audio-visual \textbf{ca}ptioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at \href{https://d-orca-llm.github.io/}{https://d-orca-llm.github.io/}. Our code, data, and checkpoints will be available at \href{https://github.com/WeChatCV/D-ORCA/}{https://github.com/WeChatCV/D-ORCA/}.

D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning

TL;DR

D-ORCA tackles dialogue-centric audio-visual captioning by introducing a specialized RL framework (GRPO) with three rewards that target speaker attribution, content fidelity, and temporal grounding. It pairs a Q-Former–assisted audio-visual backbone with LoRA-tuned LLMs, trained through SFT, pre-DPO, and GRPO on a large bilingual dataset (DVD-Train) and evaluated on a dedicated benchmark (DVD-Bench). The approach yields state-of-the-art performance among open-source models in speaker identification, ASR accuracy, and temporal localization, while maintaining competitive results on general AV QA benchmarks, despite using 8B parameters. The work also provides robust, LLM-based evaluation protocols and demonstrates the importance of data curation (dialogue-rich English/Chinese videos) for improving dialogue-aware perception in multimodal systems.

Abstract

Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a \textbf{d}ialogue-centric \textbf{o}mni-modal large language model optimized for \textbf{r}obust audio-visual \textbf{ca}ptioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at \href{https://d-orca-llm.github.io/}{https://d-orca-llm.github.io/}. Our code, data, and checkpoints will be available at \href{https://github.com/WeChatCV/D-ORCA/}{https://github.com/WeChatCV/D-ORCA/}.
Paper Structure (19 sections, 7 equations, 4 figures, 7 tables)

This paper contains 19 sections, 7 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Dialogue-centric Audio-Visual Understanding
  4. Optimization for Audio-Visual Captioning
  5. Methods
  6. Model Architecture and Training Pipeline
  7. Dialogue-centric Reward Modeling
  8. Experimental Setup
  9. Data Specifications
  10. Model, training and evaluation specifications
  11. Experimental Results
  12. Main Results
  13. Impact of pre-DPO
  14. Ablation on Reward Balancing in GRPO
  15. Robustness of LLM-based Evaluation Metrics
  16. ...and 4 more sections

Figures (4)

  • Figure 1: An example of video captions generated by different audio-visual LLMs for a dialogue-centric video clip. Green highlights indicate correct speaker attribution, red indicates incorrect attribution, and orange marks ambiguous or implicit speaker references. Existing open-source audio-visual LLMs struggle to accurately comprehend video dialogue, while our D-ORCA demonstrates more accurate and robust audio-visual understanding for dialogue-centric scenarios.
  • Figure 2: Architecture and reward computation in D-ORCA. The model processes chronologically interleaved audio–visual tokens as input. During GRPO training, an external LLM parser is used to extract predicted ASR texts, and also identify the speaker and timestamp for each reference spoken sentence guided by a candidate character list. Based on these structured outputs, dialogue-centric rewards ($r_{\text{speaker}}$, $r_{\text{content}}$, $r_{\text{time}}$) are computed to optimize the accuracy of speaker attribution, speech content, and sentence-level temporal alignment.
  • Figure 3: Distribution of video sources in English (En) and Chinese (Zh) subset of our DVD-Bench. The dataset covers a diverse range of genres, including movies, TV dramas, TV shows, and interviews.
  • Figure 4: Training curves of D-ORCA during the GRPO stage. The plots demonstrate the progressive improvement of the model across (a) speaker attribution accuracy, (b) speech content fidelity, (c) sentence-level temporal precision, and (d) the overall objective reward function.