Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models

TL;DR

This paper tackles the practical barrier of using model-agnostic explanations for expensive LLMs by proposing a budget-friendly proxy framework that leverages smaller, efficient models to approximate larger models' local decision boundaries. Central to the approach is a screen-and-apply mechanism that first statistically verifies proxy fidelity at the task level and then checks instance-level agreement before deploying proxy explanations. Across 12 LLMs and three representative tasks, the method achieves over 90% fidelity at a substantial reduction in costs (up to 88% savings in some scenarios), and demonstrates actionable utility in prompt compression and poisoned-example removal. By open-sourcing data and benchmarks (XLLM-Bench), the work enables scalable, cost-effective interpretability for LLM development with practical impact on optimization workflows.

Abstract

Post-hoc explanations provide transparency and are essential for guiding model optimization, such as prompt engineering and data sanitation. However, applying model-agnostic techniques to Large Language Models (LLMs) is hindered by prohibitive computational costs, rendering these tools dormant for real-world applications. To revitalize model-agnostic interpretability, we propose a budget-friendly proxy framework that leverages efficient models to approximate the decision boundaries of expensive LLMs. We introduce a screen-and-apply mechanism to statistically verify local alignment before deployment. Our empirical evaluation confirms that proxy explanations achieve over 90% fidelity with only 11% of the oracle's cost. Building on this foundation, we demonstrate the actionable utility of our framework in prompt compression and poisoned example removal. Results show that reliable proxy explanations effectively guide optimization, transforming interpretability from a passive observation tool into a scalable primitive for LLM development. Additionally, we open-source code and datasets to facilitate future research.

Figures (10)

• Figure 1: The dilemma of using model-agnostic explanation methods to optimize expensive LLMs.
• Figure 2: The workflow for leveraging proxy explanations from budget-friendly models to reduce the cost of explaining expensive LLMs.
• Figure 3: Accuracy of proxy LIME explanations on SST, MMLU, and Natural Questions datasets.
• Figure 4: Accuracy of proxy LIME explanations on Large Movie Review, Fake News, and Web Question datasets.
• Figure 5: The recommended few-shot template provided by OpenAI.