TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction

TL;DR

This work defines Theme-aware Video Dialogue Crafting (TVDC) and introduces TV-Dialogue, a multimodal, multi-agent framework that generates theme-aligned dialogues for videos of arbitrary length without training. By assigning theme-aware roles to sub-agents, enabling real-time perception of visual cues via a Vision-Language Model, and applying a self-correction loop, TV-Dialogue achieves strong theme alignment and visual consistency. The authors contribute the Multi-Theme Video Dialogue (MVD) dataset and a multi-granularity evaluation benchmark, demonstrating that TV-Dialogue outperforms state-of-the-art baselines and improves downstream video-text retrieval when used for data augmentation. The approach promises practical impact in video re-creation, dubbing, and other multimodal tasks, while highlighting ethical considerations around video content interpretation and potential misunderstandings under theme manipulation.

Abstract

Recent advancements in LLMs have accelerated the development of dialogue generation across text and images, yet video-based dialogue generation remains underexplored and presents unique challenges. In this paper, we introduce Theme-aware Video Dialogue Crafting (TVDC), a novel task aimed at generating new dialogues that align with video content and adhere to user-specified themes. We propose TV-Dialogue, a novel multi-modal agent framework that ensures both theme alignment (i.e., the dialogue revolves around the theme) and visual consistency (i.e., the dialogue matches the emotions and behaviors of characters in the video) by enabling real-time immersive interactions among video characters, thereby accurately understanding the video content and generating new dialogue that aligns with the given themes. To assess the generated dialogues, we present a multi-granularity evaluation benchmark with high accuracy, interpretability and reliability, demonstrating the effectiveness of TV-Dialogue on self-collected dataset over directly using existing LLMs. Extensive experiments reveal that TV-Dialogue can generate dialogues for videos of any length and any theme in a zero-shot manner without training. Our findings underscore the potential of TV-Dialogue for various applications, such as video re-creation, film dubbing and its use in downstream multimodal tasks.

Figures (5)

• Figure 1: Given an arbitrary user-specified theme, the Theme-aware Video Dialogue Crafting (TVDC) task seeks to generate novel dialogues aligned with video content and theme. The solid box represents the original dialogue, while the dashed box represents the new dialogue about the "presidential election".
• Figure 2: Overview of TV-Dialogue. The TV-Dialogue initially assigns a relevant role to each sub-agent based on the given theme and video, enabling immersive interaction among the sub-agents in the dialogue process (Stage 1). Sub-agents maintain visual consistency by perceiving video content, querying historical memory, and receiving messages from other agents, thereby generating high-quality dialogues (Stage 2). The generated dialogues undergo self-correction for further improvement (Stage 3).
• Figure 3: Videos per theme.
• Figure 4: Comparison of dialogues generated by different methods in the last-1 sentence prediction. The top-left corner represents the first frame of the dialogue in the video. Although the values of traditional metrics are very low, the generated dialogues are consistent with the video content and theme.
• Figure 5: Comparison of different themes in terms of Theme Relevance ($\mathbb{TR}$) and Scenario Consistency ($\mathbb{SC}$).