Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic

Zi'ou Zheng, Xiaodan Zhu

TL;DR

NatLogAttack leverages natural logic to craft systematic adversarial examples for natural language inference, introducing label-preserving and label-flipping attack modes to probe the reasoning capabilities of NLI models. By grounding perturbations in monotonicity and projection through the seven natural-logic relations, the framework generates high-quality candidates via WordNet substitutions, alternations with AltLM, and insertion/deletion operations, followed by rigorous quality control. Across SNLI, MNLI, MED, HELP, and SICK, NatLogAttack achieves higher human-validated attack success rates and requires fewer queries than strong baselines, highlighting vulnerabilities in current models under logic-based attacks. The work demonstrates that logic-grounded attacks can reveal genuine reasoning gaps, offering a foundation for future logic-aware adversarial training and analysis of NLI systems’ reasoning properties.

Abstract

Reasoning has been a central topic in artificial intelligence from the beginning. The recent progress made on distributed representation and neural networks continues to improve the state-of-the-art performance of natural language inference. However, it remains an open question whether the models perform real reasoning to reach their conclusions or rely on spurious correlations. Adversarial attacks have proven to be an important tool to help evaluate the Achilles' heel of the victim models. In this study, we explore the fundamental problem of developing attack models based on logic formalism. We propose NatLogAttack to perform systematic attacks centring around natural logic, a classical logic formalism that is traceable back to Aristotle's syllogism and has been closely developed for natural language inference. The proposed framework renders both label-preserving and label-flipping attacks. We show that compared to the existing attack models, NatLogAttack generates better adversarial examples with fewer visits to the victim models. The victim models are found to be more vulnerable under the label-flipping setting. NatLogAttack provides a tool to probe the existing and future NLI models' capacity from a key viewpoint and we hope more logic-based attacks will be further explored for understanding the desired property of reasoning.

NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic

TL;DR

NatLogAttack leverages natural logic to craft systematic adversarial examples for natural language inference, introducing label-preserving and label-flipping attack modes to probe the reasoning capabilities of NLI models. By grounding perturbations in monotonicity and projection through the seven natural-logic relations, the framework generates high-quality candidates via WordNet substitutions, alternations with AltLM, and insertion/deletion operations, followed by rigorous quality control. Across SNLI, MNLI, MED, HELP, and SICK, NatLogAttack achieves higher human-validated attack success rates and requires fewer queries than strong baselines, highlighting vulnerabilities in current models under logic-based attacks. The work demonstrates that logic-grounded attacks can reveal genuine reasoning gaps, offering a foundation for future logic-aware adversarial training and analysis of NLI systems’ reasoning properties.

Abstract

Reasoning has been a central topic in artificial intelligence from the beginning. The recent progress made on distributed representation and neural networks continues to improve the state-of-the-art performance of natural language inference. However, it remains an open question whether the models perform real reasoning to reach their conclusions or rely on spurious correlations. Adversarial attacks have proven to be an important tool to help evaluate the Achilles' heel of the victim models. In this study, we explore the fundamental problem of developing attack models based on logic formalism. We propose NatLogAttack to perform systematic attacks centring around natural logic, a classical logic formalism that is traceable back to Aristotle's syllogism and has been closely developed for natural language inference. The proposed framework renders both label-preserving and label-flipping attacks. We show that compared to the existing attack models, NatLogAttack generates better adversarial examples with fewer visits to the victim models. The victim models are found to be more vulnerable under the label-flipping setting. NatLogAttack provides a tool to probe the existing and future NLI models' capacity from a key viewpoint and we hope more logic-based attacks will be further explored for understanding the desired property of reasoning.
Paper Structure (28 sections, 2 equations, 2 figures, 9 tables, 1 algorithm)

This paper contains 28 sections, 2 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial Attacks in NLP.
  4. Natural Logic.
  5. NatLogAttack: A Natural-logic-based Attack Framework
  6. Background
  7. NatLogAttack Setups and Principles
  8. Generation and Quality Control
  9. Preparing Natural Logic Relations
  10. Candidate Generation
  11. Basic Word Perturbation.
  12. Enhancing Alternation.
  13. Insertion and Deletion.
  14. Attack Quality Control
  15. Iterative and Multi-rounds Attacking
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Overview of NatLogAttack generation and attacking process.
  • Figure 2: Query numbers (QNs) of attack models. Red dots are the medians of QNs of different attack models. The blue and orange shapes show the densities of query numbers for BERT and RoBERTa, respectively.