FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation

TL;DR

FitCF introduces ZeroCF and FitCF to automate high-quality counterfactual generation in NLP by tying feature attribution to counterfactual edits. ZeroCF selects important words from a fine-tuned BERT and prompts an LLM to produce a counterfactual in zero-shot; FitCF then verifies label flips and uses verified examples as demonstrations for few-shot prompting. Across AG News and SST2, FitCF outperforms baselines (Polyjuice, BAE, FIZLE) and ZeroCF, with LIME and SHAP generally yielding more faithful attributions and stronger correlations to counterfactual quality. A key finding is that attribution faithfulness strongly correlates with counterfactual quality, and the number of demonstrations has the largest impact on performance, guiding future attribution-guided generation research.

Abstract

Counterfactual examples are widely used in natural language processing (NLP) as valuable data to improve models, and in explainable artificial intelligence (XAI) to understand model behavior. The automated generation of counterfactual examples remains a challenging task even for large language models (LLMs), despite their impressive performance on many tasks. In this paper, we first introduce ZeroCF, a faithful approach for leveraging important words derived from feature attribution methods to generate counterfactual examples in a zero-shot setting. Second, we present a new framework, FitCF, which further verifies aforementioned counterfactuals by label flip verification and then inserts them as demonstrations for few-shot prompting, outperforming two state-of-the-art baselines. Through ablation studies, we identify the importance of each of FitCF's core components in improving the quality of counterfactuals, as assessed through flip rate, perplexity, and similarity measures. Furthermore, we show the effectiveness of LIME and Integrated Gradients as backbone attribution methods for FitCF and find that the number of demonstrations has the largest effect on performance. Finally, we reveal a strong correlation between the faithfulness of feature attribution scores and the quality of generated counterfactuals, which we hope will serve as an important finding for future research in this direction.

Figures (6)

• Figure 1: Given an instance $x$ from the AG News dataset classified as "sci/tech", our ZeroCF approach generates few-shot examples, whose important words are determined by LIME for a BERT model. FitCF then generates a counterfactual $\tilde{x}$ on this basis. The edits to original instance $x$ are highlighted in orange, yielding $\tilde{x}$ which is classified as "business".
• Figure 2: The upper part of the figure illustrates how counterfactuals are generated by ZeroCF using important words extracted by the explainer (BERT) through various feature important methods (Gradient, Integrated Gradients, LIME, SHAP). Lower part of the figure shows the pipeline of FitCF involving demonstrations selection, automatic construction of counterfactual examples by ZeroCF, label flip verification, and counterfactual generation.
• Figure 3: A Kendall’s tau ($\tau$) that quantifies the degree of correspondence between the ranking of generated counterfactuals' quality and the ranking of feature attribution evaluation results is reported.
• Figure 4: Visualization of clustering in AG News and SST2, where stars denote cluster centroids.
• Figure 5: Label distribution of AG News and SST2.