Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explaining Large Language Models with gSMILE

Zeinab Dehghani, Mohammed Naveed Akram, Koorosh Aslansefat, Adil Khan, Yiannis Papadopoulos

TL;DR

gSMILE tackles the opacity of large language models by presenting a model-agnostic, perturbation-based token attribution method that leverages input perturbations, OWMD/WMD distances, and a local weighted linear surrogate to generate intuitive heatmaps. It extends the SMILE framework to text prompts and evaluates across GPT-3.5-turbo-instruct, LLaMA 3.1 Instruct Turbo, and Claude 2.1 using metrics for attribution accuracy, fidelity, stability, faithfulness, and consistency. Claude 2.1 yields the strongest attribution fidelity while GPT-3.5-turbo-instruct demonstrates the greatest explanation consistency, with LLaMA 3.1 showing comparative robustness. The work advances practical interpretability for high-stakes AI by enabling human-aligned token attributions without access to model internals and by guiding prompt design and model selection for reliable explanations.

Abstract

Large Language Models (LLMs) such as GPT, LLaMA, and Claude achieve remarkable performance in text generation but remain opaque in their decision-making processes, limiting trust and accountability in high-stakes applications. We present gSMILE (generative SMILE), a model-agnostic, perturbation-based framework for token-level interpretability in LLMs. Extending the SMILE methodology, gSMILE uses controlled prompt perturbations, Wasserstein distance metrics, and weighted linear surrogates to identify input tokens with the most significant impact on the output. This process enables the generation of intuitive heatmaps that visually highlight influential tokens and reasoning paths. We evaluate gSMILE across leading LLMs (OpenAI's gpt-3.5-turbo-instruct, Meta's LLaMA 3.1 Instruct Turbo, and Anthropic's Claude 2.1) using attribution fidelity, attribution consistency, attribution stability, attribution faithfulness, and attribution accuracy as metrics. Results show that gSMILE delivers reliable human-aligned attributions, with Claude 2.1 excelling in attention fidelity and GPT-3.5 achieving the highest output consistency. These findings demonstrate gSMILE's ability to balance model performance and interpretability, enabling more transparent and trustworthy AI systems.

Explaining Large Language Models with gSMILE

TL;DR

gSMILE tackles the opacity of large language models by presenting a model-agnostic, perturbation-based token attribution method that leverages input perturbations, OWMD/WMD distances, and a local weighted linear surrogate to generate intuitive heatmaps. It extends the SMILE framework to text prompts and evaluates across GPT-3.5-turbo-instruct, LLaMA 3.1 Instruct Turbo, and Claude 2.1 using metrics for attribution accuracy, fidelity, stability, faithfulness, and consistency. Claude 2.1 yields the strongest attribution fidelity while GPT-3.5-turbo-instruct demonstrates the greatest explanation consistency, with LLaMA 3.1 showing comparative robustness. The work advances practical interpretability for high-stakes AI by enabling human-aligned token attributions without access to model internals and by guiding prompt design and model selection for reliable explanations.

Abstract

Large Language Models (LLMs) such as GPT, LLaMA, and Claude achieve remarkable performance in text generation but remain opaque in their decision-making processes, limiting trust and accountability in high-stakes applications. We present gSMILE (generative SMILE), a model-agnostic, perturbation-based framework for token-level interpretability in LLMs. Extending the SMILE methodology, gSMILE uses controlled prompt perturbations, Wasserstein distance metrics, and weighted linear surrogates to identify input tokens with the most significant impact on the output. This process enables the generation of intuitive heatmaps that visually highlight influential tokens and reasoning paths. We evaluate gSMILE across leading LLMs (OpenAI's gpt-3.5-turbo-instruct, Meta's LLaMA 3.1 Instruct Turbo, and Anthropic's Claude 2.1) using attribution fidelity, attribution consistency, attribution stability, attribution faithfulness, and attribution accuracy as metrics. Results show that gSMILE delivers reliable human-aligned attributions, with Claude 2.1 excelling in attention fidelity and GPT-3.5 achieving the highest output consistency. These findings demonstrate gSMILE's ability to balance model performance and interpretability, enabling more transparent and trustworthy AI systems.

Paper Structure

This paper contains 46 sections, 22 equations, 14 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature review
  3. Large Language Models
  4. Explainable AI (XAI)
  5. Explainable AI (XAI) for Large Language Models
  6. Problem Definition
  7. Notation and Problem Setting
  8. Setting the Stage.
  9. Input-Level Distance
  10. Output-Level Distribution Shift
  11. Weighting via Input Similarity
  12. Fitting a Local Surrogate
  13. Theoretical Justification (Lipschitz Smoothness)
  14. Shift Function Definition.
  15. Relation to Total Variation.
  16. ...and 31 more sections

Figures (14)

  • Figure 1: Overview of gSMILE interpretability process: An input prompt is processed by a transformer-based LLM, and gSMILE generates a token-level heatmap highlighting influential words contributing to the output.
  • Figure 2: Extended LLM Categories, Models, and Descriptions
  • Figure 3: Taxonomy of LIME enhancement strategies: categorized into sampling techniques, surrogate model improvements, distance parameter adjustments, and optimization approaches.
  • Figure 4: SMILE flowchart for explaining image classification aslansefat2023explaining
  • Figure 5: gSMILE flowchart for Large Language Models
  • ...and 9 more figures