METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection

TL;DR

METOR tackles open-vocabulary video visual relationship detection by jointly modeling objects and relationships in a unified, query-based framework. It introduces a CLIP-based contextual refinement encoding module to produce instance-specific text and object queries, and an iterative enhancement module to mutually refine object, object-trajectory, and relationship representations via spatio-temporal processing, all trained end-to-end with multiple contrastive and contextual losses. The approach achieves state-of-the-art results on VidVRD and VidOR, with substantial gains on novel categories, demonstrating improved generalization in open-vocabulary settings without relying on ground-truth trajectory detectors. This work offers a practical path toward robust open-vocabulary video understanding by mitigating error propagation and leveraging rich semantic context.

Abstract

Open-vocabulary video visual relationship detection aims to detect objects and their relationships in videos without being restricted by predefined object or relationship categories. Existing methods leverage the rich semantic knowledge of pre-trained vision-language models such as CLIP to identify novel categories. They typically adopt a cascaded pipeline to first detect objects and then classify relationships based on the detected objects, which may lead to error propagation and thus suboptimal performance. In this paper, we propose Mutual EnhancemenT of Objects and Relationships (METOR), a query-based unified framework to jointly model and mutually enhance object detection and relationship classification in open-vocabulary scenarios. Under this framework, we first design a CLIP-based contextual refinement encoding module that extracts visual contexts of objects and relationships to refine the encoding of text features and object queries, thus improving the generalization of encoding to novel categories. Then we propose an iterative enhancement module to alternatively enhance the representations of objects and relationships by fully exploiting their interdependence to improve recognition performance. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate that our framework achieves state-of-the-art performance.

Figures (5)

• Figure 1: (a) Existing methods adopt a cascaded pipeline. (b) Our method jointly models objects and relationships for their mutually enhancement. (c) Statistics of triplet errors on the VidVRD test set, categorized into object errors (OE) and relationship errors (RE). We also highlight the relationship errors caused by error propagation in the cascaded methods (OV-MMP and EOV-MMP), where the relationship is correctly classified using ground-truth object trajectories but misclassified using detected object trajectories.
• Figure 2: Overview of the proposed framework.
• Figure 3: An overview of the proposed contextual refinement encoding module.
• Figure 4: An overview of the proposed iterative enhancement module.
• Figure 5: Qualitative results of feature distributions via T-SNE. (a) and (b) display the feature distribution of objects before and after mutual enhancement, while (c) and (d) show the feature distribution of relationships. The categories labeled in red font represent the novel categories.