DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models

TL;DR

DiaDem tackles the underexplored challenge of accurate dialogue description in audiovisual captioning by introducing DiaDemBench for rigorous evaluation of speaker attribution and utterance transcription. It builds DiaDem through a two-stage post-training regime on a carefully constructed dataset (70K dialogue-rich captions for SFT, 15K non-dialogue captions, and 3K manually annotated GRPO samples), leveraging a difficulty-partitioned GRPO to optimize utterance transcription and speaker attribution without sacrificing general captioning quality. Empirical results show DiaDem surpasses the Gemini series in dialogue-description accuracy while maintaining competitive performance on holistic audiovisual benchmarks, underscoring the viability of dialogue-aware captioning for multimodal LLMs. The work also introduces adaptive dialogue matching and a robust benchmarking protocol, offering a strong foundation for further improvements in multi-speaker and overlapping-speech scenarios.

Abstract

Accurate dialogue description in audiovisual video captioning is crucial for downstream understanding and generation tasks. However, existing models generally struggle to produce faithful dialogue descriptions within audiovisual captions. To mitigate this limitation, we propose DiaDem, a powerful audiovisual video captioning model capable of generating captions with more precise dialogue descriptions while maintaining strong overall performance. We first synthesize a high-quality dataset for SFT, then employ a difficulty-partitioned two-stage GRPO strategy to further enhance dialogue descriptions. To enable systematic evaluation of dialogue description capabilities, we introduce DiaDemBench, a comprehensive benchmark designed to evaluate models across diverse dialogue scenarios, emphasizing both speaker attribution accuracy and utterance transcription fidelity in audiovisual captions. Extensive experiments on DiaDemBench reveal even commercial models still exhibit substantial room for improvement in dialogue-aware captioning. Notably, DiaDem not only outperforms the Gemini series in dialogue description accuracy but also achieves competitive performance on general audiovisual captioning benchmarks, demonstrating its overall effectiveness.

Figures (18)

• Figure 1: Qualitative and quantitative analysis of dialogue description capabilities in existing models, highlighting substantial room for improvement in speaker-utterance attribution. In the generated audiovisual captions, speaker attributions are color-coded as incorrect, ambiguous, and correct.
• Figure 2: Overview of the curation pipeline of DiaDemBench.
• Figure 3: When the dialogue segmentation of the ground-truth annotations is inconsistent with that of the model predicts, issues arise regardless of whether adjacent utterances from the same speaker are merged.
• Figure 4: Data annotation pipeline for SFT.
• Figure 5: On the left, we present an illustration of an audiovisual video caption with accurate dialogue descriptions generated by DiaDem, featuring both correct speaker attribution and precise utterance transcription, as well as other general audiovisual details. On the right, we showcase four representative dialogue scenarios from DiaDemBench that are commonly challenging for existing models to produce accurate dialogue descriptions, with the aim of providing insights for future advancements in audiovisual video captioning models.