Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models

Haisong Gong, Zhibo Liu, Qiang Liu, Shu Wu, Liang Wang

TL;DR

NAG addresses the problem of integrating graphs into language models without external encoders by unifying topology and semantics inside the LM. It introduces topology-aware attention and structural position calibration to enable encoder-free graph reasoning, with two implementations: NAG-Zero (zero-interference adapters) and NAG-LoRA (low-rank adapters). Through extensive experiments on synthetic topology tasks and real-world semantic graph reasoning benchmarks, NAG demonstrates robust graph comprehension and superior semantic reasoning compared to linearization and dual-path baselines. The results suggest a paradigm shift toward native graph processing within Transformer architectures, offering a simpler, coherent framework for text-graph modeling with practical efficiency benefits.

Abstract

Prevailing methods for integrating graphs into Language Models (LMs) typically rely on a segregated architecture: external Graph Neural Networks (GNNs) encode structural topology, while LMs process textual semantics. We argue this approach is suboptimal for text-graphs: it creates a conceptually disjointed interaction paradigm. By segregating structural encoding from semantic processing, these systems must perform a complex implicit alignment between abstract graph tokens and concrete textual elements. Challenging the necessity of external encoders, we propose NAG (Native Architecture for Graphs), a unified framework that internalizes graph processing within the LM's native manifold. Instead of bridging disparate embedding spaces, NAG repurposes the self-attention mechanism to enforce topological dependencies and recalibrates positional IDs to ensure structural equivalence. This allows the model to harness its intrinsic linguistic capability to simultaneously comprehend node and edge content alongside structural topology. We introduce two efficient implementations: NAG-Zero for absolute preservation of the base model's linguistic capabilities, and NAG-LoRA for enhanced structural adaptation. Experiments across diverse graph tasks validate that NAG achieves robust graph comprehension without the overhead of external encoders, offering a simpler, more coherent paradigm for text-graph modeling.

NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models

TL;DR

NAG addresses the problem of integrating graphs into language models without external encoders by unifying topology and semantics inside the LM. It introduces topology-aware attention and structural position calibration to enable encoder-free graph reasoning, with two implementations: NAG-Zero (zero-interference adapters) and NAG-LoRA (low-rank adapters). Through extensive experiments on synthetic topology tasks and real-world semantic graph reasoning benchmarks, NAG demonstrates robust graph comprehension and superior semantic reasoning compared to linearization and dual-path baselines. The results suggest a paradigm shift toward native graph processing within Transformer architectures, offering a simpler, coherent framework for text-graph modeling with practical efficiency benefits.

Abstract

Prevailing methods for integrating graphs into Language Models (LMs) typically rely on a segregated architecture: external Graph Neural Networks (GNNs) encode structural topology, while LMs process textual semantics. We argue this approach is suboptimal for text-graphs: it creates a conceptually disjointed interaction paradigm. By segregating structural encoding from semantic processing, these systems must perform a complex implicit alignment between abstract graph tokens and concrete textual elements. Challenging the necessity of external encoders, we propose NAG (Native Architecture for Graphs), a unified framework that internalizes graph processing within the LM's native manifold. Instead of bridging disparate embedding spaces, NAG repurposes the self-attention mechanism to enforce topological dependencies and recalibrates positional IDs to ensure structural equivalence. This allows the model to harness its intrinsic linguistic capability to simultaneously comprehend node and edge content alongside structural topology. We introduce two efficient implementations: NAG-Zero for absolute preservation of the base model's linguistic capabilities, and NAG-LoRA for enhanced structural adaptation. Experiments across diverse graph tasks validate that NAG achieves robust graph comprehension without the overhead of external encoders, offering a simpler, more coherent paradigm for text-graph modeling.
Paper Structure (35 sections, 11 equations, 4 figures, 5 tables)

This paper contains 35 sections, 11 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The Dual-pathway: Graph Neural Networks as Prefixes.
  4. Graph Reasoning via Textual Linearization.
  5. Graph Reasoning via Specialized Transformer.
  6. The NAG Framework
  7. Problem Formulation
  8. Input Construction via Structural Flattening
  9. Topology-Aware Attention Mechanism
  10. Structural Position Calibration
  11. Efficient Training Implementations
  12. Theoretical Intuition: Tuning the Message Function
  13. Experiments
  14. Experimental Setup and Baselines
  15. Topological Awareness
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Comparison of graph encoding paradigms: (Top) Dual-pathway architecture using an external graph encoder to generate graph tokens, which are prepended to the text sequence for the language model. (Bottom) Our proposed NAG framework, which eliminates the external encoder and models graph structure natively via a topology-aware attention mechanism and recalibrated positional indexing within the language model.
  • Figure 2: Input Construction and Attention Mechanism of NAG. (Top) The graph is flattened with recalibrated positional IDs to ensure structural equivalence. (Bottom) The topology-aware mask unifies semantic understanding and structural reasoning: 'Core / Sparse' denotes structural dependencies and the sparse query-graph interaction mode, while 'Full Extension' represents the additional attention pathways activated in the full interaction mode.
  • Figure 3: Illustration of the NAG-Zero Mechanism. Residual adapters operate exclusively on special tokens, while semantic text tokens pass through unchanged. This selective processing guarantees zero interference with the backbone LM.
  • Figure 4: Analysis of query-graph interaction strategies. We illustrate the performance trade-offs between Sparse and Full attention mechanisms under both NAG-Zero (blue) and NAG-LoRA (orange) settings.