Universal Scene Graph Generation

TL;DR

This work introduces Universal Scene Graph (USG), a cross-modal representation that unifies objects and relations across images, text, video, and 3D data. It presents USG-Par, a five-component end-to-end parser with a modality-specific encoder, a shared mask decoder, an object associator, a relation proposal constructor, and a relation decoder, augmented by a text-centric scene contrastive learning objective to mitigate domain imbalances. The approach demonstrates improved SG generation across single- and multi-modality datasets and shows robust cross-modal object alignment and out-of-domain generalization. By leveraging a modality-unified framework and cross-modal supervisory signals, USG-Par advances holistic scene understanding with potential benefits for multimodal AI systems and downstream tasks. The work also discusses limitations and future directions, including enhancing long-term video understanding and integrating external knowledge for more complex cross-modal grounding.

Abstract

Scene graph (SG) representations can neatly and efficiently describe scene semantics, which has driven sustained intensive research in SG generation. In the real world, multiple modalities often coexist, with different types, such as images, text, video, and 3D data, expressing distinct characteristics. Unfortunately, current SG research is largely confined to single-modality scene modeling, preventing the full utilization of the complementary strengths of different modality SG representations in depicting holistic scene semantics. To this end, we introduce Universal SG (USG), a novel representation capable of fully characterizing comprehensive semantic scenes from any given combination of modality inputs, encompassing modality-invariant and modality-specific scenes. Further, we tailor a niche-targeting USG parser, USG-Par, which effectively addresses two key bottlenecks of cross-modal object alignment and out-of-domain challenges. We design the USG-Par with modular architecture for end-to-end USG generation, in which we devise an object associator to relieve the modality gap for cross-modal object alignment. Further, we propose a text-centric scene contrasting learning mechanism to mitigate domain imbalances by aligning multimodal objects and relations with textual SGs. Through extensive experiments, we demonstrate that USG offers a stronger capability for expressing scene semantics than standalone SGs, and also that our USG-Par achieves higher efficacy and performance.

Figures (20)

• Figure 1: Illustrations of SGs (top) of single modalities in text, image, video, and 3D, and our proposed Universal SG (bottom). Note that the USG instance shown here is under the combination of four complete modalities, while practically any modality can be absent freely. Also, the temporal coreference edges are omitted for visual clarity (a full version is given in the Appendix).
• Figure 2: Overview of USG-Par architecture. It mainly consists of five modules, including modality-specific encoders, shared mask decoder, object associator, relation proposal constructor, and relation decoder.
• Figure 3: Illustration of the object associator for establishing associations between different modalities.
• Figure 4: Illustration of the object-level and relation-level text-centric scene contrasting learning mechanism.
• Figure 5: Comparison of multimodal tasks with and without USG integration. Baselines: multimodal relation extraction (MRE) Wu00BC23, emotion detection (ED) 10.1145/3689646, and 3D visual QA (3DVQA) AzumaMKK22.