iFlip: Iterative Feedback-driven Counterfactual Example Refinement

TL;DR

This paper tackles the challenge of generating valid counterfactual examples with large language models by moving beyond single-pass generation to an iterative refinement framework called iFlip. iFlip uses three feedback signals—confidence, feature attribution, and natural language feedback—to guide successive refinements under an explicit validity criterion, with early stopping to prevent over-editing. Empirical results show substantial improvements in counterfactual validity (average LFR up to $+57.8\%$ over baselines) and user satisfaction, and ablations demonstrate the critical roles of iteration, feedback type, and stopping rules. The approach also demonstrates effective counterfactual data augmentation, boosting model performance and robustness, though it notes higher computational costs and English-centric evaluation as limitations with promising directions for future work.

Abstract

Counterfactual examples are minimal edits to an input that alter a model's prediction. They are widely employed in explainable AI to probe model behavior and in natural language processing (NLP) to augment training data. However, generating valid counterfactuals with large language models (LLMs) remains challenging, as existing single-pass methods often fail to induce reliable label changes, neglecting LLMs' self-correction capabilities. To explore this untapped potential, we propose iFlip, an iterative refinement approach that leverages three types of feedback, including model confidence, feature attribution, and natural language. Our results show that iFlip achieves an average 57.8% higher validity than the five state-of-the-art baselines, as measured by the label flipping rate. The user study further corroborates that iFlip outperforms baselines in completeness, overall satisfaction, and feasibility. In addition, ablation studies demonstrate that three components are paramount for iFlip to generate valid counterfactuals: leveraging an appropriate number of iterations, pointing to highly attributed words, and early stopping. Finally, counterfactuals generated by iFlip enable effective counterfactual data augmentation, substantially improving model performance and robustness.

Figures (15)

• Figure 1: iFlip framework overview. Left: The core iterative loop of our method, which includes a generator $\mathcal{G}$ and an explained model $\mathcal{M}$. $\mathcal{G}$ generates and refines counterfactuals. $\mathcal{M}$ evaluates counterfactuals under our validity criterion: a counterfactual is considered valid if $\mathcal{M}$'s predicted label is equal to the target label $\tilde{y}$. Otherwise, invalid counterfactuals are iteratively refined until the validity criterion is met to avoid unnecessary iterations or the maximal number of refinement steps is reached. Right: Examples of the three types of feedback we employed for iFlip: Confidence, Feature Attribution, and Natural Language Feedback.
• Figure 2: Iterative refinement effectiveness on OLMo2-7B (AG News) across feedback signals. Left: pass@k curves up to $\mathcal{K}=15$. Right: per-iteration improvement in flip rate ($\Delta$Flip Rate).
• Figure 3: Iterative refinement with OLMo2-7B under early stopping, averaged across all datasets and feedback signals. Fail $\rightarrow$ Fail indicates no label change, Fail $\rightarrow$ Success denotes a successful label flip in the current turn, and Previous Success marks instances flipped in earlier turns.
• Figure 4: Examples from IMDb, AG News, and SNLI datasets.
• Figure 5: Label distributions across datasets.