LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations

TL;DR

LLMCheckup introduces a dialogue-driven interpretability tool that lets users interrogate any autoregressive LLM about its behavior without fine-tuning. It unifies downstream task execution, self-explanation, and response generation within a single-model workflow by parsing user intents into SQL-like operations, and it exposes a suite of white-box and black-box XAI techniques, tutorials, and multi-modal and external information capabilities. The paper details two parsing strategies (Guided Decoding and Multi-prompt Parsing), a Flask-based interface, and add-ons such as information retrieval and dialogue sharing, plus evaluations of parsing accuracy and data augmentation across several LLMs on fact-checking and commonsense QA. The work demonstrates the feasibility of conversational interpretability as a practical baseline for human-model interaction, offering a compact, transferable framework with clear directions for future enhancements and human-centered evaluation.

Abstract

Interpretability tools that offer explanations in the form of a dialogue have demonstrated their efficacy in enhancing users' understanding (Slack et al., 2023; Shen et al., 2023), as one-off explanations may fall short in providing sufficient information to the user. Current solutions for dialogue-based explanations, however, often require external tools and modules and are not easily transferable to tasks they were not designed for. With LLMCheckup, we present an easily accessible tool that allows users to chat with any state-of-the-art large language model (LLM) about its behavior. We enable LLMs to generate explanations and perform user intent recognition without fine-tuning, by connecting them with a broad spectrum of Explainable AI (XAI) methods, including white-box explainability tools such as feature attributions, and self-explanations (e.g., for rationale generation). LLM-based (self-)explanations are presented as an interactive dialogue that supports follow-up questions and generates suggestions. LLMCheckupprovides tutorials for operations available in the system, catering to individuals with varying levels of expertise in XAI and supporting multiple input modalities. We introduce a new parsing strategy that substantially enhances the user intent recognition accuracy of the LLM. Finally, we showcase LLMCheckup for the tasks of fact checking and commonsense question answering.

Figures (5)

• Figure 1: LLMCheckup dialogue with data augmentation and rationalization operations on a commonsense question answering task (ECQA). Boxes (not part of the actual UI) indicate the original instance from the dataset as well as its prediction (cyan) and the explanation requested by the user (orange).
• Figure 2: On the left, a dialogue example asking for explanation in natural language about a ECQA-like customized question. The workflow of LLMCheckup is shown on the right side.
• Figure 3: LLMCheckup interface with welcome message, free-text rationale and sample generator buttons. Expert XAI level and OPRO strategy are selected. For example multi-turn dialogues, see Table \ref{['tab:examples_1']} and Table \ref{['tab:examples_2']}.
• Figure 4: QA tutorial with different knowledge level in XAI.
• Figure 5: External information retrieval of an instance from COVID-Fact.