Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LIME-LLM: Probing Models with Fluent Counterfactuals, Not Broken Text

George Mihaila, Suleyman Olcay Polat, Poli Nemkova, Himanshu Sharma, Namratha V. Urs, Mark V. Albert

TL;DR

Empirical results demonstrate that LIME-LLM establishes a new benchmark for black-box NLP explainability, achieving significant improvements in local explanation fidelity compared to both traditional perturbation-based methods and recent generative alternatives.

Abstract

Local explanation methods such as LIME (Ribeiro et al., 2016) remain fundamental to trustworthy AI, yet their application to NLP is limited by a reliance on random token masking. These heuristic perturbations frequently generate semantically invalid, out-of-distribution inputs that weaken the fidelity of local surrogate models. While recent generative approaches such as LLiMe (Angiulli et al., 2025b) attempt to mitigate this by employing Large Language Models for neighborhood generation, they rely on unconstrained paraphrasing that introduces confounding variables, making it difficult to isolate specific feature contributions. We introduce LIME-LLM, a framework that replaces random noise with hypothesis-driven, controlled perturbations. By enforcing a strict "Single Mask-Single Sample" protocol and employing distinct neutral infill and boundary infill strategies, LIME-LLM constructs fluent, on-manifold neighborhoods that rigorously isolate feature effects. We evaluate our method against established baselines (LIME, SHAP, Integrated Gradients) and the generative LLiMe baseline across three diverse benchmarks: CoLA, SST-2, and HateXplain using human-annotated rationales as ground truth. Empirical results demonstrate that LIME-LLM establishes a new benchmark for black-box NLP explainability, achieving significant improvements in local explanation fidelity compared to both traditional perturbation-based methods and recent generative alternatives.

LIME-LLM: Probing Models with Fluent Counterfactuals, Not Broken Text

TL;DR

Empirical results demonstrate that LIME-LLM establishes a new benchmark for black-box NLP explainability, achieving significant improvements in local explanation fidelity compared to both traditional perturbation-based methods and recent generative alternatives.

Abstract

Local explanation methods such as LIME (Ribeiro et al., 2016) remain fundamental to trustworthy AI, yet their application to NLP is limited by a reliance on random token masking. These heuristic perturbations frequently generate semantically invalid, out-of-distribution inputs that weaken the fidelity of local surrogate models. While recent generative approaches such as LLiMe (Angiulli et al., 2025b) attempt to mitigate this by employing Large Language Models for neighborhood generation, they rely on unconstrained paraphrasing that introduces confounding variables, making it difficult to isolate specific feature contributions. We introduce LIME-LLM, a framework that replaces random noise with hypothesis-driven, controlled perturbations. By enforcing a strict "Single Mask-Single Sample" protocol and employing distinct neutral infill and boundary infill strategies, LIME-LLM constructs fluent, on-manifold neighborhoods that rigorously isolate feature effects. We evaluate our method against established baselines (LIME, SHAP, Integrated Gradients) and the generative LLiMe baseline across three diverse benchmarks: CoLA, SST-2, and HateXplain using human-annotated rationales as ground truth. Empirical results demonstrate that LIME-LLM establishes a new benchmark for black-box NLP explainability, achieving significant improvements in local explanation fidelity compared to both traditional perturbation-based methods and recent generative alternatives.
Paper Structure (32 sections, 3 equations, 4 figures, 6 tables)

This paper contains 32 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries and Problem Formulation
  5. The LIME-LLM Framework
  6. Hypothesis-Driven Infilling Strategies
  7. Local Surrogate Optimization
  8. Hybrid Proximity Kernel
  9. Experiments
  10. Datasets and Sampling Protocol
  11. Baseline Methods
  12. Evaluation Metrics
  13. Implementation Details
  14. Results and Discussion
  15. Alignment with Human Rationales (ROC-AUC)
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Overview of the proposed LIME-LLM framework. LIME-LLM constructs local neighborhoods using hypothesis-driven, on-manifold LLM infilling, generating one fluent sample per binary mask via label-preserving neutral or counterfactual boundary strategies. The resulting semantic neighborhood enables faithful local surrogate explanations.
  • Figure 2: Comparison of ROC and Precision-Recall (PR) curves across three evaluation datasets: CoLA (top), SST-2 (middle), and HateXplain (bottom). Shaded regions indicate 95% confidence intervals over 30 random seeds for stochastic methods. LIME-LLM consistently outperforms LIME and Partition SHAP, and achieves faithfulness comparable to Integrated Gradients while remaining fully model-agnostic.
  • Figure 3: Ablation Study 1 (Base LLM: GPT-4.1). Comparison of ROC and Precision–Recall (PR) curves across three evaluation datasets: CoLA (top), SST-2 (middle), and HateXplain (bottom). Shaded regions denote 95% confidence intervals over 30 random seeds for stochastic methods.
  • Figure 4: Ablation Study 1 (Base LLM: Gemini 3 Flash). Comparison of ROC and Precision-Recall (PR) curves across three evaluation datasets: CoLA (top), SST-2 (middle), and HateXplain (bottom). Shaded regions denote 95% confidence intervals over 30 random seeds for stochastic methods.