Adversarial Question Answering Robustness: A Multi-Level Error Analysis and Mitigation Study

TL;DR

This work analyzes adversarial robustness in extractive QA by performing a multi-level error analysis on AddSent and ELECTRA variants, uncovering negation confusion and entity substitution as the dominant failure modes. It shows that a simple adversarial training mix of 80% clean and 20% adversarial data yields strong gains, but capacity bottlenecks in small models limit generalization; scaling to ELECTRA-base Removes the robustness-accuracy trade-off and delivers substantial improvements on both clean and adversarial data. Building on this, the authors propose three mitigation strategies and find that NER-guided Entity-Aware contrastive learning provides the best overall performance, achieving AddSent EM of 89.89% and SQuAD EM of 90.73% with a 94.9% closure of the adversarial gap, i.e., near parity between clean and adversarial performance. These results underscore that model capacity and targeted linguistic-awareness training are key to robust QA, with practical implications for deploying resistant systems in real-world settings.

Abstract

Question answering (QA) systems achieve impressive performance on standard benchmarks like SQuAD, but remain vulnerable to adversarial examples. This project investigates the adversarial robustness of transformer models on the AddSent adversarial dataset through systematic experimentation across model scales and targeted mitigation strategies. We perform comprehensive multi-level error analysis using five complementary categorization schemes, identifying negation confusion and entity substitution as the primary failure modes. Through systematic evaluation of adversarial fine-tuning ratios, we identify 80% clean + 20% adversarial data as optimal. Data augmentation experiments reveal a capacity bottleneck in small models. Scaling from ELECTRA-small (14M parameters) to ELECTRA-base (110M parameters) eliminates the robustness-accuracy trade-off, achieving substantial improvements on both clean and adversarial data. We implement three targeted mitigation strategies, with Entity-Aware contrastive learning achieving best performance: 89.89% AddSent Exact Match (EM) and 90.73% SQuAD EM, representing 94.9% closure of the adversarial gap. To our knowledge, this is the first work integrating comprehensive linguistic error analysis with Named Entity Recognition (NER)-guided contrastive learning for adversarial QA, demonstrating that targeted mitigation can achieve near-parity between clean and adversarial performance.

Figures (6)

• Figure 1: Linguistic pattern distribution in baseline model errors. Negation confusion and entity substitution are the dominant failure modes, accounting for 70.3% of errors combined.
• Figure 2: Performance across 5 adversarial training ratios with ELECTRA-small. Left: AddSent (adversarial) performance peaks at 80-20 ratio (66.57%). Right: SQuAD (clean) performance degrades with higher adversarial data. The 80-20 configuration achieves optimal balance.
• Figure 3: Robustness-accuracy trade-off analysis for ELECTRA-small across 5 ratios. X-axis shows clean performance cost (SQuAD degradation), Y-axis shows adversarial performance gain (AddSent improvement). Quadrants indicate trade-off regions. The 80-20 and 90-10 ratios achieve the best balance with positive adversarial gains and acceptable clean costs.
• Figure 4: Capacity bottleneck comparison: ELECTRA-small vs ELECTRA-base across configurations. The small model plateaus with adversarial training, while base model continues improving. The divergence at "Adversarial 80-20" configuration demonstrates how scaling resolves the capacity limitation.
• Figure 5: Step-by-step improvement breakdown showing the experimental progression. Baseline → Adversarial training → Data augmentation → Model scaling. The largest gain (+21.86% AddSent) comes from scaling to ELECTRA-base, demonstrating that capacity is the key bottleneck.