Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation

Qianli Wang, Van Bach Nguyen, Yihong Liu, Fedor Splitt, Nils Feldhus, Christin Seifert, Hinrich Schütze, Sebastian Möller, Vera Schmitt

TL;DR

The paper tackles multilingual counterfactual explanations generated by large language models, evaluating directly generated counterfactuals and translation-based variants across six languages on XNLI and SIB200. It combines automatic metrics ($LFR$, $TS$, $PPL$) with cross-lingual edit similarity to assess validity and linguistic quality, and identifies four pervasive error patterns. It finds that TB-CFs often achieve higher counterfactual validity but incur heavier editing and translation artifacts, while DG-CFs show language-dependent performance; cross-lingual edits reveal common strategies among European languages but diverge for others. The study further investigates multilingual counterfactual data augmentation (CDA) and finds it generally yields better performance than cross-lingual CDA, particularly for low-resource languages, though overall gains are limited by CF imperfections. These findings illuminate the challenges and potential of multilingual explanations and suggest avenues such as improved multilingual CF quality, richer language coverage, and post-training enhancements to boost practical impact.

Abstract

Counterfactuals refer to minimally edited inputs that cause a model's prediction to change, serving as a promising approach to explaining the model's behavior. Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency. However, their effectiveness in generating multilingual counterfactuals remains unclear. To this end, we conduct a comprehensive study on multilingual counterfactuals. We first conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages. Although translation-based counterfactuals offer higher validity than their directly generated counterparts, they demand substantially more modifications and still fall short of matching the quality of the original English counterfactuals. Second, we find the patterns of edits applied to high-resource European-language counterfactuals to be remarkably similar, suggesting that cross-lingual perturbations follow common strategic principles. Third, we identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages. Finally, we reveal that multilingual counterfactual data augmentation (CDA) yields larger model performance improvements than cross-lingual CDA, especially for lower-resource languages. Yet, the imperfections of the generated counterfactuals limit gains in model performance and robustness.

Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation

TL;DR

The paper tackles multilingual counterfactual explanations generated by large language models, evaluating directly generated counterfactuals and translation-based variants across six languages on XNLI and SIB200. It combines automatic metrics (, , ) with cross-lingual edit similarity to assess validity and linguistic quality, and identifies four pervasive error patterns. It finds that TB-CFs often achieve higher counterfactual validity but incur heavier editing and translation artifacts, while DG-CFs show language-dependent performance; cross-lingual edits reveal common strategies among European languages but diverge for others. The study further investigates multilingual counterfactual data augmentation (CDA) and finds it generally yields better performance than cross-lingual CDA, particularly for low-resource languages, though overall gains are limited by CF imperfections. These findings illuminate the challenges and potential of multilingual explanations and suggest avenues such as improved multilingual CF quality, richer language coverage, and post-training enhancements to boost practical impact.

Abstract

Counterfactuals refer to minimally edited inputs that cause a model's prediction to change, serving as a promising approach to explaining the model's behavior. Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency. However, their effectiveness in generating multilingual counterfactuals remains unclear. To this end, we conduct a comprehensive study on multilingual counterfactuals. We first conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages. Although translation-based counterfactuals offer higher validity than their directly generated counterparts, they demand substantially more modifications and still fall short of matching the quality of the original English counterfactuals. Second, we find the patterns of edits applied to high-resource European-language counterfactuals to be remarkably similar, suggesting that cross-lingual perturbations follow common strategic principles. Third, we identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages. Finally, we reveal that multilingual counterfactual data augmentation (CDA) yields larger model performance improvements than cross-lingual CDA, especially for lower-resource languages. Yet, the imperfections of the generated counterfactuals limit gains in model performance and robustness.
Paper Structure (66 sections, 3 equations, 14 figures, 15 tables)

This paper contains 66 sections, 3 equations, 14 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Counterfactual Example Generation.
  4. Counterfactual Explanation Evaluation.
  5. Multilingual Counterfactuals.
  6. Experimental Setup
  7. Counterfactual Generation
  8. Datasets
  9. XNLI
  10. SIB200
  11. Language Selection
  12. Models
  13. Evaluation Setup
  14. Automatic Evaluation
  15. Label Flip Rate (LFR)
  16. ...and 51 more sections

Figures (14)

  • Figure 1: ❶ Parallel original inputs from the SIB200 dataset classified as "Politics" in English (HTML]d8d8d8en), German (HTML]E0E6FFde), and Swahili (HTML]DDF6D2sw), ❷ their corresponding counterfactuals aimed at changing the label towards "Science/Technology" (Sci/Tech), ❸ automatic evaluation results and ❹ English translations of the generated counterfactuals. Multilingual counterfactuals are evaluated using three automatic metrics (label flip$\uparrow$, perplexity$\downarrow$ and similarity$\uparrow$). In the multilingual counterfactuals and their English translations, words modified by LLMs are highlighted in HTML]CC99FFpurple.
  • Figure 2: An overview of counterfactual generation process. Given an original instance $x$ from SIB200 classified as "Travel", the corresponding counterfactual $\tilde{x}$ is classified as "Health". Edits to $x$ are underlined.
  • Figure 3: Cosine similarity scores of counterfactuals $\tilde{x}_\ell$ across different languages measured by SBERT.
  • Figure 4: (a) Copy-paste rates and (b) language confusion rates for counterfactuals across different languages.
  • Figure 5: Categorization of error types in generating multilingual counterfactuals across five languages: HTML]E5CCFFArabic, HTML]E0E6FFGerman, HTML]FFF1D5Spanish, HTML]F2CFC2Hindi, and HTML]DDF6D2Swahili. For each error type, we present two examples and their corresponding HTML]d8d8d8English translations. Error spans are marked with HTML]FF9999red highlights to indicate the exact locations of the issues.
  • ...and 9 more figures