SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses

TL;DR

SynopGround introduces a large-scale, long-form video grounding dataset comprising over $2800$ hours of TV drama footage paired with human-written synopses, each synopsis paragraph temporally localized in the corresponding episode. It formalizes Multi-Paragraph Video Grounding (MPVG) and presents the Local-Global Multimodal Reasoner (LGMR), a baseline that jointly models long-range temporal structure and cross-modal interactions via a local-global encoder and iterative decoder using a subparagraph extractor. The approach demonstrates strong performance gains over prior multi-sentence grounding methods, with extensive ablations confirming the importance of long-context modeling, multi-feature fusion (SlowFast, CLIP, OCR), and explicit cross-modal guidance. The work provides public data and code, enabling research on long-term narrative understanding and practical applications in story-aware video retrieval and editing.

Abstract

Video grounding is a fundamental problem in multimodal content understanding, aiming to localize specific natural language queries in an untrimmed video. However, current video grounding datasets merely focus on simple events and are either limited to shorter videos or brief sentences, which hinders the model from evolving toward stronger multimodal understanding capabilities. To address these limitations, we present a large-scale video grounding dataset named SynopGround, in which more than 2800 hours of videos are sourced from popular TV dramas and are paired with accurately localized human-written synopses. Each paragraph in the synopsis serves as a language query and is manually annotated with precise temporal boundaries in the long video. These paragraph queries are tightly correlated to each other and contain a wealth of abstract expressions summarizing video storylines and specific descriptions portraying event details, which enables the model to learn multimodal perception on more intricate concepts over longer context dependencies. Based on the dataset, we further introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG), which takes as input multiple paragraphs and a long video for grounding each paragraph query to its temporal interval. In addition, we propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term multimodal inputs for MPVG. Our method provides an effective baseline solution to the multi-paragraph video grounding problem. Extensive experiments verify the proposed model's effectiveness as well as its superiority in long-term multi-paragraph video grounding over prior state-of-the-arts. Dataset and code are publicly available. Project page: https://synopground.github.io/.

Figures (7)

• Figure 1: Comparison of video grounding datasets. Our SynopGround is the first dataset to introduce the challenges of both long videos and long queries into video grounding.
• Figure 2: Illustration of the proposed Multi-Paragraph Video Grounding (MPVG) problem and two representative samples in our SynopGround. Given a video and a synopsis $\mathcal{Q}$ that contains $N$ paragraphs $\{Q_1, Q_2, ..., Q_N\}$, the model should predict the corresponding temporal interval for each paragraph $Q_i$ in the form of starting and ending time.
• Figure 3: Data distribution in our dataset. (a): Genre distribution of TV dramas. (b): Normalized duration of target video segments. (c): Number of queries per video. (d): Normalized start timestamp distribution. (e): Normalized end timestamp distribution.
• Figure 4: Our proposed Local-Global Multimodal Reasoner (LGMR). It consists of a local-global temporal encoder for structural long-term temporal modeling and a local-global iterative decoder to adaptively reason through local and global semantics.
• Figure 5: Visualization results of the model predictions and ground truth for multi-paragraph video grounding in SynopGround dataset. This example is selected from the test set and the video is the 18-th episode of the TV Drama Vampire Diaries Season 1. Due to space limitation, we uniformly sample frames from the temporal interval that encloses both the model predictions and ground truth for better visual presentation. (Best viewed on screen when zoomed in)