Unpacking Robustness in Inflectional Languages: Adversarial Evaluation and Mechanistic Insights

TL;DR

The paper investigates robustness of inflectional languages to adversarial text attacks by introducing a mechanistic interpretability protocol based on Edge Attribution Patching (EAP-IG) and a parallel Inflection Circuit Detection Dataset built from MultiEmo. It evaluates four attack methods (TextFooler, TextBugger, WordNetTextFooler, BERT-Attack) across English, Polish, and Czech, revealing language- and task-dependent robustness and highlighting Inflectional morphology as a key factor. Mechanistic analysis identifies inflection-specific circuits in Polish, with Layer 0 attention heads emerging as inflection-sensitive and contributing to improved robustness under attack; English shows fewer such components due to lower morphological variability. The study highlights that inflection-aware training and circuit-level understanding can enhance adversarial resilience, while acknowledging data availability and parallel-corpus generation challenges as main limitations and pointing to future work on richer circuit analyses and dataset creation.

Abstract

Various techniques are used in the generation of adversarial examples, including methods such as TextBugger which introduce minor, hardly visible perturbations to words leading to changes in model behaviour. Another class of techniques involves substituting words with their synonyms in a way that preserves the text's meaning but alters its predicted class, with TextFooler being a prominent example of such attacks. Most adversarial example generation methods are developed and evaluated primarily on non-inflectional languages, typically English. In this work, we evaluate and explain how adversarial attacks perform in inflectional languages. To explain the impact of inflection on model behaviour and its robustness under attack, we designed a novel protocol inspired by mechanistic interpretability, based on Edge Attribution Patching (EAP) method. The proposed evaluation protocol relies on parallel task-specific corpora that include both inflected and syncretic variants of texts in two languages -- Polish and English. To analyse the models and explain the relationship between inflection and adversarial robustness, we create a new benchmark based on task-oriented dataset MultiEmo, enabling the identification of mechanistic inflection-related elements of circuits within the model and analyse their behaviour under attack.

Figures (4)

• Figure 1: Distribution of adversarial examples based on similarity to the original data for four evaluation metrics for the AG_News dataset.
• Figure 2: Distribution of adversarial examples based on similarity to the original data for four evaluation metrics for the CSFD dataset.
• Figure 3: Inflection circuit detection methodology with adversarial evaluation of identified circuits.
• Figure 4: Graph comparison of Polish syncretic vs. Polish inflectional circuits, showing the 50 most important edges. Nodes represent model submodules: rectangular (present in syncretic), dotted (removed in inflectional), and triangular (added in inflexional). Edge thickness reflects connection importance.