NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality

TL;DR

The paper tackles the challenge of fine-grained video-language compositionality, where temporal dynamics complicate understanding of actions, attributes, relations, and objects. It introduces AARO, a comprehensive benchmark for assessing VidL models on four compositional categories, and NAVERO, a training framework that leverages diverse, offline negative-text augmentations and negative-augmented vision-text losses to improve fine-grained reasoning. NAVERO demonstrates significant gains over baselines on AARO datasets and maintains strong text-video retrieval performance, while also showing strong transfer to image-text compositional benchmarks such as VL-Checklist and COCO. The approach contributes to more robust cross-modal understanding in video-language tasks and reveals promising generalization to image-text domains with reduced supervision and data requirements.

Abstract

We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. Composition understanding becomes particularly challenging for video data since the compositional relations rapidly change over time in videos. We first build a benchmark named AARO to evaluate composition understanding related to actions on top of spatial concepts. The benchmark is constructed by generating negative texts with incorrect action descriptions for a given video and the model is expected to pair a positive text with its corresponding video. Furthermore, we propose a training method called NAVERO which utilizes video-text data augmented with negative texts to enhance composition understanding. We also develop a negative-augmented visual-language matching loss which is used explicitly to benefit from the generated negative text. We compare NAVERO with other state-of-the-art methods in terms of compositional understanding as well as video-text retrieval performance. NAVERO achieves significant improvement over other methods for both video-language and image-language composition understanding, while maintaining strong performance on traditional text-video retrieval tasks.

Figures (6)

• Figure 1: Video-Language compositional reasoning challenges the models to capture the subtle semantics in the captions. Given the paired video, the existing video-language models are confused in distinguishing the true caption from the corrupted captions, while NAVERO makes a clear distinction.
• Figure 2: Illustration of the method NAVERO. We design a mixed-type multi-round action-augmented negative text generator to create diverse negative texts. A new negative-augmented vision-text matching loss is proposed to boost the fine-grained compositionality learning.
• Figure 3: Example of a video-text pair where the text description is corrupted in a rule-based method and an LLM-based method. The suffixes '1r' and '5r' denote the number of rounds of calls to the negative text generator per sample. More rounds lead to more extensive set of variations from the original text.
• Figure 4: Bar plot of accuracy $acc$ on each compositional type of evaluation on the video-text and image-text compositionality with the video-text model trained on different number of frames. Results are reported on AARO-MSRVTT and VL-Checklist dataset.
• Figure 5: Case study for training on the Vanilla and NAVERO model on the AARO-MSRVTT. We provide test examples which are incorrectly predicted by the vanilla video-text model but are correctly predicted by the NAVERO. The score difference $(p(T|V) - p(T^{neg}|V))$ from Vanilla to NAVERO are reported, where $T$, $T^{neg}$ and $V$ denote the original text, negative text and the video, respectively.