LMExplainer: Grounding Knowledge and Explaining Language Models

TL;DR

LMExplainer tackles the opacity of language models by grounding their reasoning in a large knowledge graph and explaining with a graph-attention-based surrogate. It combines key element extraction, element-graph interpretation via graph attention networks, and an instruction-driven explanation generator, augmented by an LM debugger to assess faithfulness and guide improvements. Evaluations on CommonsenseQA and OpenBookQA show LMExplainer achieving competitive or superior accuracy against strong KG-augmented baselines, and providing more comprehensive explanations than prior methods. The approach reduces hallucinations through grounded knowledge and offers actionable insights for LM development, contributing to more transparent, reliable, and equitable AI.

Abstract

Language models (LMs) like GPT-4 are important in AI applications, but their opaque decision-making process reduces user trust, especially in safety-critical areas. We introduce LMExplainer, a novel knowledge-grounded explainer that clarifies the reasoning process of LMs through intuitive, human-understandable explanations. By leveraging a graph attention network (GAT) with a large-scale knowledge graph (KG), LMExplainer not only precisely narrows the reasoning space to focus on the most relevant knowledge but also grounds its reasoning in structured, verifiable knowledge to reduce hallucinations and enhance interpretability. LMExplainer effectively generates human-understandable explanations to enhance transparency and streamline the decision-making process. Additionally, by incorporating debugging into the explanation, it offers expertise suggestions that improve LMs from a developmental perspective. Thus, LMExplainer stands as an enhancement in making LMs more accessible and understandable to users. We evaluate LMExplainer on benchmark datasets such as CommonsenseQA and OpenBookQA, demonstrating that it outperforms most existing methods. By comparing the explanations generated by LMExplainer with those of other models, we show that our approach offers more comprehensive and clearer explanations of the reasoning process. LMExplainer provides a deeper understanding of the inner workings of LMs, advancing towards more reliable, transparent, and equitable AI.

Figures (10)

• Figure 1: The LMExplainer architecture. LMExplainer addresses the lack of transparency in black-box LMs by providing interpretable explanations for their reasoning behavior. Given an input context, LMExplainer retrieves relevant knowledge from a KG, integrates it with LM embeddings to construct a surrogate subgraph, and interprets it using graph attention to identify key reason-elements contributing to the LM's decision. Based on these elements, LMExplainer generates explanations clarifying why the LM chose the predicted answer and ruled out alternatives. The LM Debugger further enhances user trust by evaluating the faithfulness, completeness, minimality, and accuracy of the generated explanations.
• Figure 2: Example of LMExplainer's explanation for a correct LM prediction, with high scores and potential enhancement suggestions from the LM Debugger on key dimensions.
• Figure 3: Simulated reasoning scores split by LM's prediction correctness.
• Figure 4: The instructions for explanation generators.
• Figure 5: The instructions for LM Debugger.