SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation

TL;DR

This work tackles the challenging problem of manipulating scene graphs in a coherent, commonsense-aware manner. SG-Tailor reframes node additions and edge changes as autoregressive inter-object relationship predictions, using a Cut-And-Stitch strategy to ensure conflict-free edits. It encodes triplets as quintuple tokens and trains a decoder-only Transformer to predict next tokens, achieving robust, globally consistent graph manipulation. The approach delivers substantial improvements over baselines on multiple 3D scene benchmarks and proves effective as a plug-in for scene generation and robotic manipulation, with strong qualitative and user-study support.

Abstract

Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.

Figures (8)

• Figure 1: SG-Tailor for scene graph manipulation. SG-Tailor manipulates a given scene graph in two modes: (a) Node Addition and (b) Edge Change. For node addition, SG-Tailor autoregressively reasons commonsense relationships between a newly introduced node and existing nodes (e.g., The wardrobe node should be near the bed node and to the left of the chair node, as the chair is already left of the bed.). For edge change, it maintains the desired edge while resolving conflicts (e.g., naively moving the chair node to the left of the bed node causes a table conflict. SG-Tailor resolves this by replacing the conflicting edge to maintain coherence).
• Figure 2: Training and Inference. Starting from A. Scene Graph$G$, we convert $N$ triplets into a set of B. Quintuple Tokens$Q$, resulting in $5N$ tokens. Each token $q_i \in Q$ is then combined with special tokens to form token sequences $t_i$. During C. Training, the model $\phi$ learns to perform next-token prediction, including both object nodes and relationships, with the mask attention mechanism. This process runs until it reaches the sequence-end token $\mathbf{[EOS]}$. During D. Inference, $\phi$ accepts all existing tokens $\left\{t_1,...t_N\right\}$ from the given graph and query tokens $t_{N+1}^*$ containing each two of nodes and special tokens to perform next-relationship prediction. The predicted relationship is integrated into $t_{N+1}^*$, forming $t_{N+1}$. In this way, the model autoregressively reasons about inter-object relationships.
• Figure 3: Qualitative comparison. We assess the quality of scene graph manipulation across different methods by generating the corresponding scenes using the Graph-to-3D model. dhamo2021graph.
• Figure 4: Qualitative comparison of SG-Bot w/ and w/o SG-Tailor. We show three examples of SG-Tailor facilitating the robotic manipulation tasks. More examples can be found in the supplementary.
• Figure 5: Statistics of the user study in the manipulation task