Generalized Visual Relation Detection with Diffusion Models

TL;DR

This paper tackles the limitation of VRD models restricted to predefined relation categories by introducing Diff-VRD, a diffusion-model-based framework that treats visual relations as continuous embeddings and generates relation sequences conditioned on subject–object cues via cross-attention with CLIP-based features. It adds embedding and rounding steps to connect discrete relation vocabularies to latent embeddings, enabling open-vocabulary and generalized VRD, and employs a ground-truth–guided bipartite matching with auxiliary supervision to map relations to pairs. To evaluate generalized VRD beyond annotated categories, two proxy metrics are proposed: text-to-image retrieval and SPICE-based precision–recall curves, showing that Diff-VRD achieves competitive VRD recall while producing richer, more diverse relations. Experiments on HOI and scene graph generation benchmarks confirm that the diffusion-based approach can match or surpass state-of-the-art discriminative VRD methods and substantially enhance relation diversity and open-vocabulary capability, with practical implications for open-world scene understanding.

Abstract

Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image. Although recent VRD models have achieved impressive performance, they are all restricted to pre-defined relation categories, while failing to consider the semantic ambiguity characteristic of visual relations. Unlike objects, the appearance of visual relations is always subtle and can be described by multiple predicate words from different perspectives, e.g., ``ride'' can be depicted as ``race'' and ``sit on'', from the sports and spatial position views, respectively. To this end, we propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner, termed Diff-VRD. We model the diffusion process in a latent space and generate all possible relations in the image as an embedding sequence. During the generation, the visual and text embeddings of subject-object pairs serve as conditional signals and are injected via cross-attention. After the generation, we design a subsequent matching stage to assign the relation words to subject-object pairs by considering their semantic similarities. Benefiting from the diffusion-based generative process, our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets. To properly evaluate this generalized VRD task, we introduce two evaluation metrics, i.e., text-to-image retrieval and SPICE PR Curve inspired by image captioning. Extensive experiments in both human-object interaction (HOI) detection and scene graph generation (SGG) benchmarks attest to the superiority and effectiveness of Diff-VRD.

Figures (7)

• Figure 1: An example of visual relation detection (VRD). (a) Existing methods achieve VRD through discriminative classification. (b) We propose to generate visual relations via a diffusion model, conditioned on visual features of subject-object pairs. We model all visual relations collectively as one sequence but display only the relations of one subject-object pair for clarity.
• Figure 2: Overview of Diff-VRD. (a): Given an image, we first detect all the objects with their categories and enumerate the subject-object pairs. (b): We extend the diffusion-denoising process with an embedding step $q_\phi(\bm{x}_0 | \bm{v})$ and a rounding step $p_\theta(\bm{v} | \bm{x}_0)$. We introduce a tailored training objective for this extended diffusion-denoising process, as described in Sec. \ref{['sec_vlb']}. (c): The parameter $\theta$ in the denoising process is implemented by a Transformer decoder $f_\theta$. The conditional signal includes CLIP encoded visual and text embeddings of subject-object pairs, which are concatenated as $\bm{y}$, projected by $\tau_\theta$, and injected via cross-attention layers (cf. Eq. (\ref{['eq:crossattn']})). (d): After generating the relation sequence $\bm{v}$, we assign each $v_i$ to a subject-object pair by using multi-round bipartite matching based on their semantic similarities. Our contributions are highlighted in light blue.
• Figure 3: SPICE anderson2016spice precision-recall curve (PR curve) on HICO-DET chao2018learning and V-COCO gupta2015visual datasets. We used the official caption annotations for COCO and the generated captions from OFA wang2022ofa for HICO-DET. The VRD predictions are ranked by their confidence score for SPICE precision and recall evaluation.
• Figure 4: Relation diversity evaluation on HICO-DET chao2018learning test set. Each relation category is a predicate-object combination. The x-axis represents category IDs. The y-axis denotes the total number of relation instances of each category. For a clear visualization, we show only the top 100 categories where at least one of the compared methods hits a ground-truth instance. We also counted the total number of categories predicted by each method, shown at the bottom.
• Figure 5: Ablation studies by diversity evaluation for the number of relations ($K$) of each subject-object pair, where Diff-VRD serves as an enhancement on IEtrans zhang2022fine. The experiments were conducted on VG krishna2017visual test set. We used the additionally annotated predicates from zhang2022fine as the ground-truth labels, covering more categories than the original VG dataset.