GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

TL;DR

GreaseLM addresses the challenge of answering questions requiring both contextual reasoning and external world knowledge by fusing pretrained language models with knowledge graphs across multiple layers. Its core innovation is a cross-modal fusion mechanism using an interaction token and an interaction node, enabling bidirectional information flow between the LM and KG representations in successive layers. Empirical results on CommonsenseQA, OpenBookQA, and MedQA-USMLE show consistent improvements over strong baselines and other LM+KG methods, including when using smaller, comparably sized models. The approach also demonstrates enhanced handling of linguistic nuances such as negation and hedging, and proves adaptable across domains with different language models and knowledge graphs.

Abstract

Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.

Figures (2)

• Figure 1: GreaseLM Architecture. The textual context is appended with a special interaction token and passed through $N$ LM-based unimodal encoding layers. Simultaneously, a local KG of relevant knowledge is extracted and connected to an interaction node. In the later GreaseLM layers, the language representation continues to be updated through LM layers and the KG is processed using a GNN, simulating reasoning over its knowledge. In each layer, after each modality's representation is updated, the representations of the interaction token and node are pulled, concatenated, and passed through a modality interaction (MInt) unit to mix their representations. In subsequent layers, the mixed information from the interaction elements mixes with their respective modalities, allowing knowledge from the KG to affect the representations of individual tokens, and context from language to affect fine-grained entity knowledge representations in the GNN.
• Figure 2: Qualitative analysis of GreaseLM's graph attention weight changes across multiple layers of message passing compared with QA-GNN. GreaseLM demonstrates attention change patterns that more closely resemble the expected change in focus on the "bug" entity.