Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations

Shahin Atakishiyev, Housam K. B. Babiker, Jiayi Dai, Nawshad Farruque, Teruaki Hayashi, Nafisa Sadaf Hriti, Md Abed Rahman, Iain Smith, Mi-Young Kim, Osmar R. Zaïane, Randy Goebel

TL;DR

This paper addresses the opacity and hallucination risks of large language models by examining two complementary explanatory approaches: local explainability and mechanistic interpretability. It surveys existing methods, and presents empirical insights from healthcare and autonomous driving to explore how explanations affect trust and decision-making in safety-critical contexts. The authors identify key properties, evaluation strategies, and stress-test requirements for trustworthy explanations, and propose eight guiding principles and future directions to align LLM explanations with human users. The work emphasizes that faithful, context-aware explanations are essential for responsible deployment and offers a roadmap for improving transparency, safety, and accountability in LLM-supported tasks.

Abstract

Large language models have exhibited impressive performance across a broad range of downstream tasks in natural language processing. However, how a language model predicts the next token and generates content is not generally understandable by humans. Furthermore, these models often make errors in prediction and reasoning, known as hallucinations. These errors underscore the urgent need to better understand and interpret the intricate inner workings of language models and how they generate predictive outputs. Motivated by this gap, this paper investigates local explainability and mechanistic interpretability within Transformer-based large language models to foster trust in such models. In this regard, our paper aims to make three key contributions. First, we present a review of local explainability and mechanistic interpretability approaches and insights from relevant studies in the literature. Furthermore, we describe experimental studies on explainability and reasoning with large language models in two critical domains -- healthcare and autonomous driving -- and analyze the trust implications of such explanations for explanation receivers. Finally, we summarize current unaddressed issues in the evolving landscape of LLM explainability and outline the opportunities, critical challenges, and future directions toward generating human-aligned, trustworthy LLM explanations.

Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations

TL;DR

This paper addresses the opacity and hallucination risks of large language models by examining two complementary explanatory approaches: local explainability and mechanistic interpretability. It surveys existing methods, and presents empirical insights from healthcare and autonomous driving to explore how explanations affect trust and decision-making in safety-critical contexts. The authors identify key properties, evaluation strategies, and stress-test requirements for trustworthy explanations, and propose eight guiding principles and future directions to align LLM explanations with human users. The work emphasizes that faithful, context-aware explanations are essential for responsible deployment and offers a roadmap for improving transparency, safety, and accountability in LLM-supported tasks.

Abstract

Large language models have exhibited impressive performance across a broad range of downstream tasks in natural language processing. However, how a language model predicts the next token and generates content is not generally understandable by humans. Furthermore, these models often make errors in prediction and reasoning, known as hallucinations. These errors underscore the urgent need to better understand and interpret the intricate inner workings of language models and how they generate predictive outputs. Motivated by this gap, this paper investigates local explainability and mechanistic interpretability within Transformer-based large language models to foster trust in such models. In this regard, our paper aims to make three key contributions. First, we present a review of local explainability and mechanistic interpretability approaches and insights from relevant studies in the literature. Furthermore, we describe experimental studies on explainability and reasoning with large language models in two critical domains -- healthcare and autonomous driving -- and analyze the trust implications of such explanations for explanation receivers. Finally, we summarize current unaddressed issues in the evolving landscape of LLM explainability and outline the opportunities, critical challenges, and future directions toward generating human-aligned, trustworthy LLM explanations.
Paper Structure (31 sections, 7 equations, 10 figures)

This paper contains 31 sections, 7 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Structure of the Paper
  4. Background of Language Models
  5. The Rise of LLMs
  6. The Transformer architecture
  7. The Transformer architecture
  8. Related Work
  9. Explainability within LLMs
  10. LLMs meet XAI
  11. Local explainability of LLMs
  12. Mechanistic interpretability of LLMs
  13. Critical Epistemic Properties of LLMs for Generating Trustworthy Explanations
  14. LLM explanations for facts, beliefs, and the grey zone
  15. Tacit and explicit explanations by LLMs: Show vs tell
  16. ...and 16 more sections

Figures (10)

  • Figure 1: Local explainability and mechanistic interpretability approaches in LLMs.
  • Figure 2: The architecture of Transformer. Source: vaswani2017attention.
  • Figure 3: An example of an induction circuit provided by Rai et al. rai2024practical, based on Elhage et al.'s elhage2021mathematical findings, showing outputs of two attention heads as nodes, and link between them as edges, enabling the model to generalize from repeated patterns.
  • Figure 6: ChatGPT provides a diagnostic prediction and the corresponding evidence when provided with a radiology report on Crohn's disease. The prompt was tested on October 31st, 2024.
  • Figure 7: An example of a serious hallucination of GPT 3.5 in clinical diagnosis on the Med-HALT benchmark: Source: pal2023med.
  • ...and 5 more figures