Towards Flexible Visual Relationship Segmentation

TL;DR

FleVRS tackles the need for unified visual relationship segmentation that covers standard, promptable, and open-vocabulary tasks. It introduces a single-stage Transformer-based model with a dual-query system that decodes <subject, predicate, object> triplets with subject/object masks, while supporting structured prompts and CLIP-based open-vocabulary grounding; masks are derived from SAM to enable pixel-level supervision and Hungarian matching guides training. The approach achieves strong performance across HOI segmentation, VRD, and open-vocabulary scenarios, with notable gains on HICO-DET (+1.9 $mAP$), VRD (+$11.4$ $Acc$), and unseen HICO-DET (+$4.7$ $mAP$). The work demonstrates a significant step toward a flexible, scalable, and intuitive framework for visual understanding of relationships, with practical impact on downstream tasks requiring fine-grained relational reasoning and open-world generalization.

Abstract

Visual relationship understanding has been studied separately in human-object interaction(HOI) detection, scene graph generation(SGG), and referring relationships(RR) tasks. Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner. In this work, we propose FleVRS, a single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmentation to adapt to novel scenarios. FleVRS leverages the synergy between text and image modalities, to ground various types of relationships from images and use textual features from vision-language models to visual conceptual understanding. Empirical validation across various datasets demonstrates that our framework outperforms existing models in standard, promptable, and open-vocabulary tasks, e.g., +1.9 $mAP$ on HICO-DET, +11.4 $Acc$ on VRD, +4.7 $mAP$ on unseen HICO-DET. Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.

Figures (8)

• Figure 1: FleVRS is a single model trained to support standard, promptable and open-vocabulary fine-grained visual relationship segmentation (<subject mask, relationship categories, object mask>). It can take images only or images with structured prompts as inputs, and segment all existing relationships or the ones subject to the text prompts.
• Figure 2: Examples of converting HOI detection boxes to masks. We filter out low-quality masks during training by computing IoU between the mask and box.
• Figure 3: Overview of FleVRS. In standard VRS, without textual queries, the latent queries perform self- and cross-attention within the relationship decoder to output a triplet for each query. For promptable VRS, the decoder additionally incorporates textual queries $\mathbf{Q}^{\mathbf{t}}$, concatenated with $\mathbf{Q}^{\mathbf{v}}$. This setup similarly predicts triplets, each based on $\mathbf{Q}^{\mathbf{v}}$ outputs aligned with features from the optional textual prompt $\mathbf{Q}^{\mathbf{t}}$.
• Figure 4: Qualitative results of promptable VRS on HICO-DET hicodet test set. We show visualizations of subject and object masks and relationship category outputs, given three types of text prompts. In (c), we show the predicted predicates in bold characters. Unseen objects and predicates are denoted in red characters.
• Figure 5: Qualitative results of promptable and open-vocabulary VRS on HICO-DET hicodet test set. We show visualizations of the predicted triplet with the highest matching score, including subject, object masks, and predicted predicate categories. There are three types of textual prompts shown in (a), (b), and (c), with unseen concepts in the rightmost columns. In (c), we show the predicted predicates in bold characters. Unseen objects and predicates are denoted in red characters. Note that the subject is always "person" in HICO-DET.