SemPool: Simple, robust, and interpretable KG pooling for enhancing language models

TL;DR

KG-powered QA often relies on GNNs that propagate information over a graph, but performance degrades when critical answer information is missing. SemPool replaces complex graph propagation with a simple, LM-centric pooling of KG semantics: each KG fact is verbalized and encoded by the same seed LM, edge embeddings are globally pooled with a self-attention mechanism to yield a graph representation $g_q$, and this semantic vector is fused into the LM at one or more transformer layers. The approach yields robustness to graph perturbations and consistently outperforms state-of-the-art GNN baselines by about $2.27$ percentage points in the adversarial setting where answer information is absent, across three QA domains and multiple LMs; it also provides interpretability by revealing how different layers weigh KG facts. The key contributions are (i) a lightweight, principled semantic pooling scheme for KG facts, (ii) flexible early or late fusion of KG semantics into LM reasoning, and (iii) extensive ablations showing the importance of semantic alignment, fusion strategy, and graph perturbation robustness for KG-grounded QA, with practical impact for scalable QA systems.

Abstract

Knowledge Graph (KG) powered question answering (QA) performs complex reasoning over language semantics as well as knowledge facts. Graph Neural Networks (GNNs) learn to aggregate information from the underlying KG, which is combined with Language Models (LMs) for effective reasoning with the given question. However, GNN-based methods for QA rely on the graph information of the candidate answer nodes, which limits their effectiveness in more challenging settings where critical answer information is not included in the KG. We propose a simple graph pooling approach that learns useful semantics of the KG that can aid the LM's reasoning and that its effectiveness is robust under graph perturbations. Our method, termed SemPool, represents KG facts with pre-trained LMs, learns to aggregate their semantic information, and fuses it at different layers of the LM. Our experimental results show that SemPool outperforms state-of-the-art GNN-based methods by 2.27% accuracy points on average when answer information is missing from the KG. In addition, SemPool offers interpretability on what type of graph information is fused at different LM layers.

Figures (6)

• Figure 1: Our SemPool method performs simple graph pooling to enhance the LM's reasoning. Facts of the KG are represented by their semantic information with pre-trained LMs. SemPool aggregates the graph's semantic information into a single representation that is fed into the LM for QA.
• Figure 2: Setting when critical answer information is removed from the KG. Originally, GNNs propagate information from the answer nodes (light-red color) to other nodes of the graph, and the candidate answer with more links is more likely to be the correct answer wang2021gsc. If we remove the answer node's edges, information propagation becomes challenging and GNNs struggle to discriminate between correct and incorrect answers (Section \ref{['sec:findings']}).
• Figure 3: SemPool architecture with early (left) and late (right) fusion. Number $K$ represents the number of late fusion layers.
• Figure 4: Dev set performance with respect to the number $K$ of fusion layers, using two different LMs.
• Figure 5: Working mechanism of SemPool: Top-3 scored facts at each $K \in \{0,1,2\}$, along with their attention weights.