Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era

Xuansheng Wu, Haiyan Zhao, Yaochen Zhu, Yucheng Shi, Fan Yang, Lijie Hu, Tianming Liu, Xiaoming Zhai, Wenlin Yao, Jundong Li, Mengnan Du, Ninghao Liu

TL;DR

The paper defines Usable XAI for the LLM era and argues that explanations should both diagnose and improve LLM-based systems while also leveraging LLMs to make XAI more usable. It surveys 10 strategies spanning attribution methods, internal and latent representations, sample-based explanations, prompting, data augmentation, and human-emulation to enhance explainability and trustworthiness in LLMs. Through case studies and open-source code, it demonstrates practical pathways to debug, align, and safely deploy LLMs, while candidly outlining challenges in efficiency, faithfulness, retrieval, and ethics. The work aims to bridge the gap between explanation and practical impact, enabling more reliable, transparent, and controllable LLM applications across high-stakes domains.

Abstract

Explainable AI (XAI) refers to techniques that provide human-understandable insights into the workings of AI models. Recently, the focus of XAI is being extended toward explaining Large Language Models (LLMs). This extension calls for a significant transformation in the XAI methodologies for two reasons. First, many existing XAI methods cannot be directly applied to LLMs due to their complexity and advanced capabilities. Second, as LLMs are increasingly deployed in diverse applications, the role of XAI shifts from merely opening the ``black box'' to actively enhancing the productivity and applicability of LLMs in real-world settings. Meanwhile, the conversation and generation abilities of LLMs can reciprocally enhance XAI. Therefore, in this paper, we introduce Usable XAI in the context of LLMs by analyzing (1) how XAI can explain and improve LLM-based AI systems and (2) how XAI techniques can be improved by using LLMs. We introduce 10 strategies, introducing the key techniques for each and discussing their associated challenges. We also provide case studies to demonstrate how to obtain and leverage explanations. The code used in this paper can be found at: https://github.com/JacksonWuxs/UsableXAI_LLM.

Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era

TL;DR

The paper defines Usable XAI for the LLM era and argues that explanations should both diagnose and improve LLM-based systems while also leveraging LLMs to make XAI more usable. It surveys 10 strategies spanning attribution methods, internal and latent representations, sample-based explanations, prompting, data augmentation, and human-emulation to enhance explainability and trustworthiness in LLMs. Through case studies and open-source code, it demonstrates practical pathways to debug, align, and safely deploy LLMs, while candidly outlining challenges in efficiency, faithfulness, retrieval, and ethics. The work aims to bridge the gap between explanation and practical impact, enabling more reliable, transparent, and controllable LLM applications across high-stakes domains.

Abstract

Explainable AI (XAI) refers to techniques that provide human-understandable insights into the workings of AI models. Recently, the focus of XAI is being extended toward explaining Large Language Models (LLMs). This extension calls for a significant transformation in the XAI methodologies for two reasons. First, many existing XAI methods cannot be directly applied to LLMs due to their complexity and advanced capabilities. Second, as LLMs are increasingly deployed in diverse applications, the role of XAI shifts from merely opening the ``black box'' to actively enhancing the productivity and applicability of LLMs in real-world settings. Meanwhile, the conversation and generation abilities of LLMs can reciprocally enhance XAI. Therefore, in this paper, we introduce Usable XAI in the context of LLMs by analyzing (1) how XAI can explain and improve LLM-based AI systems and (2) how XAI techniques can be improved by using LLMs. We introduce 10 strategies, introducing the key techniques for each and discussing their associated challenges. We also provide case studies to demonstrate how to obtain and leverage explanations. The code used in this paper can be found at: https://github.com/JacksonWuxs/UsableXAI_LLM.
Paper Structure (82 sections, 10 equations, 5 figures, 3 tables)

This paper contains 82 sections, 10 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. LLM Diagnosis and Enhancement via Attribution Methods
  3. Attribution Methods for LLMs
  4. Attribution Methods for Classification
  5. Attribution Methods for Generation
  6. Usability of Attribution Methods for LLMs
  7. Challenges
  8. LLM Diagnosis and Enhancement via Interpreting Model Internals
  9. Explanation of Model Components
  10. Linear Relation between Components
  11. Importance Analysis on Single Component
  12. Circuit Analysis on Multiple Components
  13. Usability of Component Analysis
  14. Explanation of Latent Representations
  15. Non-Training based Methods
  16. ...and 67 more sections

Figures (5)

  • Figure 1: The contributions and outline of this paper. We define Usable XAI in the context of LLMs with seven strategies of enhancing LLMs with XAI, and three strategies of enhancing XAI with LLMs.
  • Figure 2: Left: An example of attribution saliency maps. Right: The pipeline of making use of attribution results to improve LLMs.
  • Figure 3: Taxonomy of explanation methods for LLM internals, including explaining model components and hidden representations.
  • Figure 4: Taxonomy of explanation techniques for trustworthiness aspects of LLMs.
  • Figure 5: CoT Prompts for Multi-hop QA.