Pair then Relation: Pair-Net for Panoptic Scene Graph Generation

TL;DR

This work targets Panoptic Scene Graph Generation (PSG) by identifying inter-object pair recalls as a bottleneck and proposing a Pair-Net framework that first learns sparse subject–object pairs with a Pair Proposal Network and a Matrix Learner, then predicts relations via a Relation Fusion module. The method combines a Mask2Former-based panoptic segmentation backbone, a learned sparse pairing mechanism, and a DETR-style relation decoder, achieving substantial improvements over prior PSG methods (notably over 10 percentage points in key Recall metrics). Key contributions include the explicit pair-wise learning objective, the Matrix Learner for sparsity, and comprehensive ablation/visualization analyses that underscore pair recall as a critical driver for PSG performance. The approach provides a strong, scalable baseline for PSG with practical implications for downstream vision tasks requiring fine-grained scene understanding and relational reasoning.

Abstract

Panoptic Scene Graph (PSG) is a challenging task in Scene Graph Generation (SGG) that aims to create a more comprehensive scene graph representation using panoptic segmentation instead of boxes. Compared to SGG, PSG has several challenging problems: pixel-level segment outputs and full relationship exploration (It also considers thing and stuff relation). Thus, current PSG methods have limited performance, which hinders downstream tasks or applications. The goal of this work aims to design a novel and strong baseline for PSG. To achieve that, we first conduct an in-depth analysis to identify the bottleneck of the current PSG models, finding that inter-object pair-wise recall is a crucial factor that was ignored by previous PSG methods. Based on this and the recent query-based frameworks, we present a novel framework: Pair then Relation (Pair-Net), which uses a Pair Proposal Network (PPN) to learn and filter sparse pair-wise relationships between subjects and objects. Moreover, we also observed the sparse nature of object pairs for both Motivated by this, we design a lightweight Matrix Learner within the PPN, which directly learns pair-wised relationships for pair proposal generation. Through extensive ablation and analysis, our approach significantly improves upon leveraging the segmenter solid baseline. Notably, our method achieves over 10\% absolute gains compared to our baseline, PSGFormer. The code of this paper is publicly available at https://github.com/king159/Pair-Net.

Figures (7)

• Figure 1: An illustration of Panoptic Scene Graph (PSG) task, framework, and performance comparisons. (a) The Panoptic Scene Graph (PSG) task involves generating object-background relations and their masks. (b) Frameworks compared include PSGTR psg, PSGFormer psg, and SGTR sgtr. Our Pair-Net uses the Pair Proposal Network (PPN) to learn object query pairs first and then extract relations between targeted subjects and objects. (c) Performance comparison shows significant improvement over previous methods, demonstrating the effectiveness of Pair-Net.
• Figure 2: An illustration of our proposed Pair-Net. It mainly contains three parts: (a) Panoptic Segmentation Network uses a query-based object segmenter to generate panoptic segmentation and object queries. (b) Pair Proposal Network generates subject-object pairs from object queries, with Matrix Learner to ensure the sparsity property. (c) The relation Fusion module models the interaction between pair-wised queries and relation queries and predicts final relation labels.
• Figure 3: Relation classes distribution of PSG and VG-150. Following bi-graph, we summarize the proportion and number of different classes in the form of head, body, tail of PSG in (a) and (c). We provide results of VG-150 in (b) and (d) for reference. The figure shows the long-tail effect on the distribution of relation classes.
• Figure 4: Illustration of Hungarian matching process before the losses calculation for one ground truth example. We define the matching cost as the sum of subject, object, and relation classification loss between the model output and the ground truth. The model outputs are the probabilities of all possible classes, and we illustrate the label of the highest value of the distribution for simplicity.
• Figure 5: The visualization of Pair Proposal Matrix. Left to right: self-multiplication of object query $\mathbf{Q}_\text{obj}\cdot \mathbf{Q}_\text{obj}^{\top}$, $\mathbf{M}_{\text{rough}}$, $\mathbf{M}_{\text{filtered}}$, and $\mathbf{M}_{\text{gt}}$. It reflects that the pairing process in PPN is not based solely on semantic similarity and shows the necessity of the PPN.