SPAN: Learning Similarity between Scene Graphs and Images with Transformers

TL;DR

SPAN proposes a Transformer-based framework to quantify similarity between scene graphs and images by mapping both modalities into a shared latent space. It introduces a novel graph serialization with structural encodings to enable graph Transformers, and a contras tive objective to align scene graphs with their corresponding images. A graph-oriented evaluation metric, R-Precision, is proposed to assess how well generated scene graphs align with image content, and new benchmarks on Visual Genome and Open Images validate the approach. The results demonstrate that SPAN yields robust image retrieval and provides a generic scene graph encoder suitable for downstream tasks, with ablations confirming the importance of structural encodings, node shuffle, and balanced Transformer depths.

Abstract

Learning similarity between scene graphs and images aims to estimate a similarity score given a scene graph and an image. There is currently no research dedicated to this task, although it is critical for scene graph generation and downstream applications. Scene graph generation is conventionally evaluated by Recall$@K$ and mean Recall$@K$, which measure the ratio of predicted triplets that appear in the human-labeled triplet set. However, such triplet-oriented metrics fail to demonstrate the overall semantic difference between a scene graph and an image and are sensitive to annotation bias and noise. Using generated scene graphs in the downstream applications is therefore limited. To address this issue, for the first time, we propose a Scene graPh-imAge coNtrastive learning framework, SPAN, that can measure the similarity between scene graphs and images. Our novel framework consists of a graph Transformer and an image Transformer to align scene graphs and their corresponding images in the shared latent space. We introduce a novel graph serialization technique that transforms a scene graph into a sequence with structural encodings. Based on our framework, we propose R-Precision measuring image retrieval accuracy as a new evaluation metric for scene graph generation. We establish new benchmarks on the Visual Genome and Open Images datasets. Extensive experiments are conducted to verify the effectiveness of SPAN, which shows great potential as a scene graph encoder.

Figures (17)

• Figure 1: Our method can quantitatively measure the scene graph-image similarity. The ground truth and generated scene graphs consist of different triplets with a low Recall$@50$, but both represent a street scene. Our method assigns higher similarity scores to the street scene image compared to the irrelevant images.
• Figure 2: Overview of our method: SPAN is a novel contrastive learning framework consisting of a scene graph Transformer and an image Transformer to reason about scene graph and image representations, respectively. We use a contrastive loss to align the scene graph and its corresponding image in the shared latent space. In order to enable the Transformer to understand the scene graph data, we propose a graph serialization approach and introduce a graph structural encoding. The node representation, edge representation, node encoding, graph embedding $\bm{G}$, and image embedding $\bm{I}$ are all learnable embeddings.
• Figure 3: To avoid potential biases resulting from the consistent assignment of node encodings to specific node representations during training, we randomly shuffle the nodes and assign them different structural encodings during each iteration.
• Figure 4: In order to maintain consistency between scene graphs and images, input images with different aspect ratios in the batch are zero-padded to a square of length $L$, rather than being cropped and resized. As a result, the length of resultant sequences is $L^2$. Zero-padded patches are masked off in the image Transformer.
• Figure 5: Architecture of the graph Transformer and the image Transformer. The learnable graph embedding in the graph sequence from the last encoder layer is adopted as the final scene graph representation. This is also for images.