OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog

TL;DR

OLViT tackles multi-turn video-grounded dialog by introducing a two-stream state-tracking approach that maintains a global dialog state across turns. An Object State Tracker attends to the most relevant objects while a Language State Tracker preserves salient linguistic references, with their outputs fused via a configurable combiner before an encoder/decoder core, enabling end-to-end training and integration with LLMs. Empirical results on DVD and SIMMC 2.1 show state-of-the-art performance in both discriminative and generative settings, with ablations confirming the complementary value of OST and LST and the benefits of pre-trained language models. This work advances robust, bias-resistant multi-modal reasoning for real-world video-dialog systems and suggests broad applicability across datasets and downstream tasks.

Abstract

We present the Object Language Video Transformer (OLViT) - a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.

Figures (7)

• Figure 1: $\mathbb{OLV}$i$\mathbb{T}$ outperforms strong baselines and achieves new state-of-the-art results on DVD (classification) and SIMMC 2.1 (generation).
• Figure 2: Architecture overview of our $\mathbb{OLV}$i$\mathbb{T}$ model. It uses MONet and DistilRoBERTa-Base to generate the object embeddings for each frame and the text embeddings, respectively. Then, we add position encoding and append the special [CLS] token. Finally, we combine the object and language state vectors of the current $i-$th turn ($s_o^{(i)}$ and $s_l^{(i)}$) with the rest of the input, which will be processed by the subsequent transformer layers. In the generative setting, a decoder block is added to predict the answer token auto-regressively.
• Figure 3: Overview of the different variants of our combiner.
• Figure 4: Performance comparison of $\mathbb{OLV}$i$\mathbb{T}$ with different combiners and state tracker variants.
• Figure 5: Performance comparison of $\mathbb{OLV}$i$\mathbb{T}$ with different numbers of objects and history turns.