V$^2$Dial: Unification of Video and Visual Dialog via Multimodal Experts

TL;DR

V$^2$Dial introduces a fully generative, multimodal expert-based framework that unifies image and video dialog by learning disentangled spatial and temporal features through dedicated experts and aligning them with a pre-trained LLM. The model uses three-stage training with spatial-temporal contrastive and matching losses, followed by LLM-alignment and end-to-end fine-tuning, and it achieves state-of-the-art results on AVSD and VisDial in zero-shot and fine-tuning settings. It also provides the first systematic study of domain shift between video and visual dialog and shows that joint training on both modalities yields the best cross-task performance. The approach demonstrates strong generalization and flexibility, enabling robust multimodal dialog across both image and video inputs with interpretable expert routing.

Abstract

We present V$^2$Dial - a novel expert-based model specifically geared towards simultaneously handling image and video input data for multimodal conversational tasks. Current multimodal models primarily focus on simpler tasks (e.g., VQA, VideoQA, video-text retrieval) and often neglect the more challenging conversational counterparts, such as video and visual/image dialog. Moreover, works on both conversational tasks evolved separately from each other despite their apparent similarities limiting their applicability potential. To this end, we propose to unify both tasks using a single model that for the first time jointly learns the spatial and temporal features of images and videos by routing them through dedicated experts and aligns them using matching and contrastive learning techniques. Furthermore, we systemically study the domain shift between the two tasks by investigating whether and to what extent these seemingly related tasks can mutually benefit from their respective training data. Extensive evaluations on the widely used video and visual dialog datasets of AVSD and VisDial show that our model achieves new state-of-the-art results across four benchmarks both in zero-shot and fine-tuning settings.

Figures (9)

• Figure 1: V$^2$Dial uses multimodal experts and outperforms state-of-the-art methods on both video and visual dialog in zero-shot and fine-tuning evaluation settings.
• Figure 2: Architectural overview ofV$^2$Dial . We adopt a training strategy composed of three stages. First, we only train the multimodal expert layers using spatial-temporal and video/image text matching losses ($\mathcal{L}_\textrm{stm}, \mathcal{L}_\textrm{vtm}$), spatial-temporal and video/image contrastive learning losses ($\mathcal{L}_\textrm{stc}, \mathcal{L}_\textrm{vtc}$), and masked language modeling loss ($\mathcal{L}_\textrm{mlm}$). Second, we couple the expert layers with a frozen pre-trained LLM end-to-end, using a generative loss $\mathcal{L}_\textrm{gen}$ to align their hidden representations. Finally, we additionally fine-tune the LLM weights on the downstream benchmarks. Each expert is a feed-forward network (FFN) composed of two fully connected layers.
• Figure 3: Overview of the training and evaluation pipeline ofV$^2$Dial . We show the different datasets used to train our model at each stage. Evaluations are conducted on the most popular video and visual dialog datasets of AVSD and VisDial, respectively. ( = video data, = image data, = closed / visual captioning data, = dialog data).
• Figure 4: Instead of training a dedicated NSP head, we propose a ranking scheme based on the cosine similarity of the candidate answers' embeddings with the respect to those of the generated ones. We used RoBERTa$_\texttt{large}$roberta and OpenAI Text Embedding-3$_\texttt{large}$ to generate these embeddings.
• Figure 5: Zero-shot qualitative examples of V$^2$Dial before and after fine-tuning on VisDial and AVSD. The former teaches the model to answer question with brief responses whereas the latter teaches it to produce longer and more elaborate answers.