MaskCaptioner: Learning to Jointly Segment and Caption Object Trajectories in Videos

TL;DR

MaskCaptioner tackles DVOC by generating synthetic, object-level captions to enable end-to-end training that jointly detects, segments, tracks, and captions all object trajectories in videos. It builds LVISCap and LV-VISCap by leveraging a state-of-the-art VLM with a multi-modal prompting strategy, and trains a unified model based on OVFormer with a BLIP-2 captioning head to produce (mask, caption) pairs for trajectories. The approach achieves state-of-the-art results on VidSTG, VLN, and BenSMOT, while enabling segmentation output for DVOC and demonstrating strong data-efficiency when pretraining on synthetic datasets. The work demonstrates the value of synthetic, localized vision-language supervision for complex video understanding and offers a scalable path toward end-to-end DVOC with segmentation.

Abstract

Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language. Due to the complexity of the task and the high cost associated with manual annotation, previous approaches resort to disjoint training strategies, potentially leading to suboptimal performance. To circumvent this issue, we propose to generate captions about spatio-temporally localized entities leveraging a state-of-the-art VLM. By extending the LVIS and LV-VIS datasets with our synthetic captions (LVISCap and LV-VISCap), we train MaskCaptioner, an end-to-end model capable of jointly detecting, segmenting, tracking and captioning object trajectories. Moreover, with pretraining on LVISCap and LV-VISCap, MaskCaptioner achieves state-of-the-art DVOC results on three existing benchmarks, VidSTG, VLN and BenSMOT. The datasets and code are available at https://www.gabriel.fiastre.fr/maskcaptioner/.

Figures (10)

• Figure 1: Examples of synthetic captions in our LV-VISCap dataset.
• Figure 1: Impact of the prompting strategy on caption quality. Scores are given by an expert human evaluator from 0 to 2 (incorrect, partially correct, or correct) on a subset from the LV-VIS validation set, and brought to 0-100 range. For the mask visual prompt experiments, we use our best prompt with either the object's bounding boxes or center point coordinates as a localization cue in the text prompt.
• Figure 2: Our MaskCaptioner data annotation pipeline: For each object, we extract its bounding boxes $\mathcal{B}_j$ from its annotated masks $\mathcal{M}_j$, and draw them in the video. The video with drawn bounding boxes $\hat{x}_j$, along with a prompt ($p_s, p_u(j)$) including additional information such as the label of the object to caption $\mathcal{C}_j$, the labels and bounding boxes of other objects in the image, is fed to a state-of-the-art VLM (Gemini 2.0 Flash team2023gemini) to generate the object-level caption. $p_s$ denotes the static prompt with general instructions and $p_u(j)$ the dynamic prompt with annotation information. We use this pipeline to generate the LVISCap and LV-VISCap datasets, used to train MaskCaptioner.
• Figure 3: Our MaskCaptioner architecture jointly segments and captions objects in videos. For each clip of $T$ frames, we obtain $N_{\textrm{obj}}$ clip-level object queries through Mask2Former cheng2021mask2formervideoinstancesegmentation, and yield associated score, mask and box predictions. At a video-level, we match the predicted object queries with the previously processed clips with the Hungarian bipartite matching algorithm, and perform instance tracking zhu2024clip. For each track, we sample $T_{\textrm{agg}}$ clips uniformly across the video, and aggregate the tracked queries to obtain a single video query per object, which we can feed to the LLM captioning head li2023blip, producing a single caption per trajectory.
• Figure 4: Impact of the generated data scale on the CapA metric. We train MaskCaptioner on a varying percentage of LVISCap captions and finetune on LV-VISCap.