Grounding is All You Need? Dual Temporal Grounding for Video Dialog

TL;DR

The Dual Temporal Grounding-enhanced Video Dialog model (DTGVD), strategically designed to merge the strengths of both dominant approaches, is introduced, emphasizes dual temporal relationships by predicting dialog turn-specific temporal regions, filtering video content accordingly, and grounding responses in both video and dialog contexts.

Abstract

In the realm of video dialog response generation, the understanding of video content and the temporal nuances of conversation history are paramount. While a segment of current research leans heavily on large-scale pretrained visual-language models and often overlooks temporal dynamics, another delves deep into spatial-temporal relationships within videos but demands intricate object trajectory pre-extractions and sidelines dialog temporal dynamics. This paper introduces the Dual Temporal Grounding-enhanced Video Dialog model (DTGVD), strategically designed to merge the strengths of both dominant approaches. It emphasizes dual temporal relationships by predicting dialog turn-specific temporal regions, filtering video content accordingly, and grounding responses in both video and dialog contexts. One standout feature of DTGVD is its heightened attention to chronological interplay. By recognizing and acting upon the dependencies between different dialog turns, it captures more nuanced conversational dynamics. To further bolster the alignment between video and dialog temporal dynamics, we've implemented a list-wise contrastive learning strategy. Within this framework, accurately grounded turn-clip pairings are designated as positive samples, while less precise pairings are categorized as negative. This refined classification is then funneled into our holistic end-to-end response generation mechanism. Evaluations using AVSD@DSTC-7 and AVSD@DSTC-8 datasets underscore the superiority of our methodology.

Figures (7)

• Figure 1: Given a video clip and dialog history (Q1$\&$A1-Q8$\&$A8), video dialog model generates the corresponding answer (A9) to the current question (Q9). Most previous methods merely exploit the nearest several turns of question-answer pairs (e.g., Q7$\&$A7, Q8$\&$A8) and Full Moment. In our method, we ground the temporal region of each QA pair in the video, and select Informative Moment and informative QA pairs for generating the responses (e.g., Q4$\&$A4, Q5$\&$A5).
• Figure 2: The pipeline of our proposed DTGVD is made up of four primary components including Basic Encoder, Temporal Grounding, Answer Generation and Contrastive Selection. The whole DTGVD model is trained with a contrastive learning-based loss function and a text generation loss function. The symbol $\oplus$ means concatenating multi-model features along the time/sequence dimension.
• Figure 3: The CIDEr performance of DTGVD and baseline (UniVL) with regard to different number of existing history turns and different length of predicted video region.
• Figure 4: The distribution of temporal regions within the training set's ground truth and the predicted regions from the test set.
• Figure 5: Qualitative results of DTGVD on AVSD@DSTC-7 dataset. The QA turns selected with green check marks are actually used by the model, which means DTGVD utilizes Q2&A2, Q3&A3 and Q4&A4, and UniVL utilizes Q4&A4, Q5&A5 and Q6&A6. The video clips framed in green are actually used by DTGVD, while the whole video is used by UniVL.