Prompt-Counterfactual Explanations for Generative AI System Behavior

TL;DR

This work tackles the problem of interpreting generative AI behavior by developing prompt-counterfactual explanations (PCEs) that relate input prompts to downstream-classified output characteristics. It adapts counterfactual explanations to non-deterministic, text-generating systems by leveraging downstream classifiers to reveal focal output traits and by handling variability through aggregate scoring. The authors present the PCE-1 algorithm, including two-phase scoring and explanation construction, and demonstrate three case studies on political bias, toxicity, and sentiment, showing how PCEs can guide bias mitigation, red-teaming, and prompt refinement. The framework lays a foundation for prompt-focused interpretability in generative AI, with practical implications for prompt engineering, safety auditing, and regulatory transparency, while outlining future research directions such as personalization and multi-modal extensions.

Abstract

As generative AI systems become integrated into real-world applications, organizations increasingly need to be able to understand and interpret their behavior. In particular, decision-makers need to understand what causes generative AI systems to exhibit specific output characteristics. Within this general topic, this paper examines a key question: what is it about the input -- the prompt -- that causes an LLM-based generative AI system to produce output that exhibits specific characteristics, such as toxicity, negative sentiment, or political bias. To examine this question, we adapt a common technique from the Explainable AI literature: counterfactual explanations. We explain why traditional counterfactual explanations cannot be applied directly to generative AI systems, due to several differences in how generative AI systems function. We then propose a flexible framework that adapts counterfactual explanations to non-deterministic, generative AI systems in scenarios where downstream classifiers can reveal key characteristics of their outputs. Based on this framework, we introduce an algorithm for generating prompt-counterfactual explanations (PCEs). Finally, we demonstrate the production of counterfactual explanations for generative AI systems with three case studies, examining different output characteristics (viz., political leaning, toxicity, and sentiment). The case studies further show that PCEs can streamline prompt engineering to suppress undesirable output characteristics and can enhance red-teaming efforts to uncover additional prompts that elicit undesirable outputs. Ultimately, this work lays a foundation for prompt-focused interpretability in generative AI: a capability that will become indispensable as these models are entrusted with higher-stakes tasks and subject to emerging regulatory requirements for transparency and accountability.

Figures (5)

• Figure 1: Downstream classification workflow for generative AI systems, which is the focus of the PCEs. Grouping the generative AI system and the downstream classifier applied to its output allows the explanation system to examine changes to the input prompt that alter the downstream classification (score).
• Figure 2: Comparison of generative AI outputs under different prompts.
• Figure 4: Frequency distribution of all words for LLaMa and OLMo. Most words appear in only one explanation, but in each case we see at least 50 words that occur across multiple explanations.
• Figure 5: How often are certain words part of the explanations, relative to their occurrence in the prompts. All of these words occur in at least two different explanations.
• Figure 7: Frequency distribution of all words occurring in explanations of toxicity for LLama and OLMo.