Graph is a Substrate Across Data Modalities

TL;DR

This work reframes graph structure as a persistent intermediate substrate that can be reused across heterogeneous modalities and tasks. It introduces G-Substrate, comprising a unified graph state space $\mathcal{G}_s$ to enforce structural compatibility and an interleaved role-based training strategy that exposes graphs to multiple functional roles. Through experiments across graph algorithmic reasoning, molecular graph description, scene graph generation, and event relation extraction, the approach consistently outperforms task-isolated and naive multi-task baselines, with the strongest gains when both structural alignment and cross-role reuse are combined. The results suggest that assembling and training graphs as reusable relational substrate improves cross-domain transfer, data efficiency, and generalization in structured AI systems, while also highlighting limitations and directions for future work in modality composition and role balancing.

Abstract

Graphs provide a natural representation of relational structure that arises across diverse domains. Despite this ubiquity, graph structure is typically learned in a modality- and task-isolated manner, where graph representations are constructed within individual task contexts and discarded thereafter. As a result, structural regularities across modalities and tasks are repeatedly reconstructed rather than accumulated at the level of intermediate graph representations. This motivates a representation-learning question: how should graph structure be organized so that it can persist and accumulate across heterogeneous modalities and tasks? We adopt a representation-centric perspective in which graph structure is treated as a structural substrate that persists across learning contexts. To instantiate this perspective, we propose G-Substrate, a graph substrate framework that organizes learning around shared graph structures. G-Substrate comprises two complementary mechanisms: a unified structural schema that ensures compatibility among graph representations across heterogeneous modalities and tasks, and an interleaved role-based training strategy that exposes the same graph structure to multiple functional roles during learning. Experiments across multiple domains, modalities, and tasks show that G-Substrate outperforms task-isolated and naive multi-task learning methods.

Figures (6)

• Figure 1: Task-isolated graph modeling vs. graph structure as a substrate. (a) Graph structure is learned in task-isolated pipelines, causing structurally similar graph patterns to occupy separate representation regions and limit cross-modal interaction. (b) We organize graph structure as a shared substrate, encouraging graph patterns from different data modalities to converge and align, so structurally analogous configurations can mutually shape the representation and improve performance.
• Figure 2: Analogous constraint roles of hub motifs across tasks. In the event graph, the hub event received participates in multiple temporal dependencies; in the scene graph, the hub object horse participates in multiple spatial relations. The central node coordinates multiple relations and constrains their joint consistency.
• Figure 3: Unified graph substrate and cross-role training. Graph structures from heterogeneous modalities are mapped into a unified graph state space $\mathcal{G}_s$, where graphs serve as persistent structural representations (b). Under naive multi-task training (c), graphs remain confined to fixed task roles, and the same graph is not reused across functional contexts. Our interleaved role-based paradigm (d) exposes the same graph $g \in \mathcal{G}_s$ to both structure-generation and structure-understanding roles, creating cross-role supervision. This role switching induces structural consistency and supports reusable graph representations across tasks and modalities.
• Figure 4: Contribution of different interleaving supervision types. Metrics are averaged accuracy for GAR, BLEU-4 for MGD, PCIs R@50 for SGG, and macro-averaged F1 for ERE.
• Figure 5: Effect of structural correctness of reused graph. Performance change ($\Delta$ vs. unified multi-task) for structurally correct and incorrect graphs. Metrics are averaged accuracy for GAR, BLEU-4 for MGD, PCIs R@50 for SGG, and macro-averaged F1 for ERE.