Panoptic Scene Graph Generation with Semantics-Prototype Learning

TL;DR

This work tackles biased predicate annotations in Panoptic Scene Graph Generation by introducing ADTrans, an adaptive data-transfer framework guided by semantics-prototype learning. It builds an unbiased predicate embedding space through invariant representation training, dynamic prototypes, and multistage data filtration, enabling more informative and consistent label transfer. The approach transfers indistinguishable triplets and potentially positive samples to improve long-tail predicate learning, validated across Visual Genome and PSG datasets with state-of-the-art gains. Practically, ADTrans enhances PSG systems' reliability and applicability to vision-language tasks by producing more coherent and accurate scene graphs.

Abstract

Panoptic Scene Graph Generation (PSG) parses objects and predicts their relationships (predicate) to connect human language and visual scenes. However, different language preferences of annotators and semantic overlaps between predicates lead to biased predicate annotations in the dataset, i.e. different predicates for same object pairs. Biased predicate annotations make PSG models struggle in constructing a clear decision plane among predicates, which greatly hinders the real application of PSG models. To address the intrinsic bias above, we propose a novel framework named ADTrans to adaptively transfer biased predicate annotations to informative and unified ones. To promise consistency and accuracy during the transfer process, we propose to measure the invariance of representations in each predicate class, and learn unbiased prototypes of predicates with different intensities. Meanwhile, we continuously measure the distribution changes between each presentation and its prototype, and constantly screen potential biased data. Finally, with the unbiased predicate-prototype representation embedding space, biased annotations are easily identified. Experiments show that ADTrans significantly improves the performance of benchmark models, achieving a new state-of-the-art performance, and shows great generalization and effectiveness on multiple datasets.

Figures (5)

• Figure 1: (a) Exemplar panoptic segmentation results of an input image. (b) present annotation transfer process. Our proposed method promotes the original dataset (annotations in red) by identifying biased annotation and potentially positive samples, and then adaptively and accurately transferring them to target triplet pairs (annotation in green).
• Figure 2: Illustration of the overall pipeline. It learns unbiased semantics-prototypes and the learned prototypes help to promise the consistency during data transfer process.
• Figure 3: R@100 for predicates under SGDet task among plain PSGTR, and PSGTR with ADTrans. ADTrans achieves more balanced and effective predicate discrimination among predicates with different frequencies than plain PSGTR (The horizontal axis, moving from left to right, illustrates predicates arranged in order of high frequency to low frequency).
• Figure 4: Visualization of plain PSGTR model and PSGTR equipped with our ADTrans. PSGTR with ADTrans can predict relationships between instances with greater accuracy and also select predicates that better match the visual scene.
• Figure 5: Visualization of the original dataset and a new dataset enhanced by our ADTRans. For the same image, we visualize its original biased annotations, and new annotations enhanced by our method. The enhanced dataset shows more informative annotations than the original one. These informative annotations promise the consistent training of models.