EGTR: Extracting Graph from Transformer for Scene Graph Generation

TL;DR

EGTR addresses the challenge of efficient Scene Graph Generation by extracting a relation graph from the by-products of a DETR-based one-stage object detector. It constructs relation representations from attention queries and keys across all layers, aggregates them with a gating mechanism, and predicts a dense relation graph with a lightweight head. An adaptive smoothing curriculum aligns relation supervision with object-detection quality, and a connectivity-prediction auxiliary task further regularizes learning. On Visual Genome and Open Images V6, EGTR achieves state-of-the-art object detection performance, competitive triplet prediction, and fast inference with a small parameter footprint, illustrating practical scalability for real-world scene understanding.

Abstract

Scene Graph Generation (SGG) is a challenging task of detecting objects and predicting relationships between objects. After DETR was developed, one-stage SGG models based on a one-stage object detector have been actively studied. However, complex modeling is used to predict the relationship between objects, and the inherent relationship between object queries learned in the multi-head self-attention of the object detector has been neglected. We propose a lightweight one-stage SGG model that extracts the relation graph from the various relationships learned in the multi-head self-attention layers of the DETR decoder. By fully utilizing the self-attention by-products, the relation graph can be extracted effectively with a shallow relation extraction head. Considering the dependency of the relation extraction task on the object detection task, we propose a novel relation smoothing technique that adjusts the relation label adaptively according to the quality of the detected objects. By the relation smoothing, the model is trained according to the continuous curriculum that focuses on object detection task at the beginning of training and performs multi-task learning as the object detection performance gradually improves. Furthermore, we propose a connectivity prediction task that predicts whether a relation exists between object pairs as an auxiliary task of the relation extraction. We demonstrate the effectiveness and efficiency of our method for the Visual Genome and Open Image V6 datasets. Our code is publicly available at https://github.com/naver-ai/egtr.

Figures (8)

• Figure 1: Motivation. SGG task aims to predict scene graph (\ref{['fig:intro2']}) with objects as nodes and relations as edges. We draw a plausible attention graph (\ref{['fig:intro3']}) simply by connecting objects with high attention weights to edges from the self-attention layers of the pre-trained DETR. It shows the potential for the self-attention from the object detector to contain rich information that aids in predicting the relations of the scene graph.
• Figure 1: The comparison of predicate category distribution based on graph regions. We compare the predicate categories of GT, hard negatives, and hard non-matchings for the validation dataset using histograms. We sort the predicate categories based on their frequency in the training dataset.
• Figure 2: Comparison with existing one-stage SGG models. (a) Object-Triplet Detection Models introduce additional triplet queries and a triplet detector to the object detector. The triplet detector requires additional modules to incorporate information from the object detector into the triplet queries. (b) Triplet Detection Models focus on detecting triplets directly without an object detector. Objects without relations may not be detected. (c) Relation Extraction Models extract relations from the object detector without a separate triplet detector. In particular, ours extracts relations more effectively by utilizing by-products from the self-attention of the object detector.
• Figure 2: Aggregated gate values from all layers. We aggregate the $N \times N$ shaped gate matrices for each layer and report the average over the entire validation dataset.
• Figure 3: The overall architecture of EGTR. We present a novel lightweight relation extractor, EGTR, which fully utilizes the self-attention of the DETR decoder. We extract query and key representations from each self-attention layer and concatenate them pairwise to represent relations between them. Additionally, we leverage the last hidden representation in same manner. To effectively aggregate information, we apply a gated sum and then predict relation with a shallow relation head.