Unbiased Scene Graph Generation by Type-Aware Message Passing on Heterogeneous and Dual Graphs

TL;DR

TA-HDG achieves improvements in the metrics of R@K and mR@K, which proves that TA-HDG can accurately predict the tail class while maintaining the competitive performance of the head class.

Abstract

Although great progress has been made in the research of unbiased scene graph generation, issues still hinder improving the predictive performance of both head and tail classes. An unbiased scene graph generation (TA-HDG) is proposed to address these issues. For modeling interactive and non-interactive relations, the Interactive Graph Construction is proposed to model the dependence of relations on objects by combining heterogeneous and dual graph, when modeling relations between multiple objects. It also implements a subject-object pair selection strategy to reduce meaningless edges. Moreover, the Type-Aware Message Passing enhances the understanding of complex interactions by capturing intra- and inter-type context in the Intra-Type and Inter-Type stages. The Intra-Type stage captures the semantic context of inter-relaitons and inter-objects. On this basis, the Inter-Type stage captures the context between objects and relations for interactive and non-interactive relations, respectively. Experiments on two datasets show that TA-HDG achieves improvements in the metrics of R@K and mR@K, which proves that TA-HDG can accurately predict the tail class while maintaining the competitive performance of the head class.

Figures (10)

• Figure 1: (a) Input image. (b) Ground true scene graph. (c) Prediction by existing SGG method. (d) Prediction by proposed SGG method. The round boxes represent objects, solid lines represent relations, dashed lines represent the direction of message passing, the red solid lines represent right subject-object pairs, and red words represent prediction errors.
• Figure 2: The architecture of TA-HDG. First, object proposals are obtained by Faster R-CNN. Second, HDGC introduces object information (distance, confidence, and existence information) to select subject-object pairs for constructing an initial graph. Then it constructs a heterogeneous graph to model the interactions between objects and constructs a dual graph to model the interactions between relations. Third, TAMP refines the features of objects and relations via the Intra-Type Message Passing on the dual graph and the Inter-Type Message Passing on the heterogeneous graph. Finally, a scene graph is generated based on the features of objects and relations.
• Figure 3: The Intra-Type Message Passing refines features via relations-relations and objects-objects on the dual graph.
• Figure 4: The process of Inter-Type Message Passing refines features via objects-relations and relations-objects on the heterogeneous graph.
• Figure 5: Results of mR@100 on the head, body, and tail classes in SGDet task. The $x$-axis is various methods, the $y$-axis is mR@100. The left is the head performance, the middle is the body performance, the right is the tail performance.