Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not-in-Perspective: Towards Shielding Google's Perspective API Against Adversarial Negation Attacks

Michail S. Alexiou, J. Sukarno Mertoguno

TL;DR

The paper tackles the vulnerability of Google's Perspective API to negation-based adversarial attacks in toxicity detection. It introduces a formal reasoning wrapper that operates as both pre-processing and post-processing around ML-based toxicity detectors, explicitly handling negation segments and exploring antonym substitution and paraphrase-based augmentation. The hybrid approaches M1.2-P and M1.5-P emerge as the strongest performers, delivering improved accuracy on Perspective-based negated test sets and reducing reported toxicity for many sentences, while Perspective remains a strong baseline. The work demonstrates that combining formal reasoning with statistical toxicity detectors yields more robust toxicity scoring, with practical implications for safer content moderation on social platforms.

Abstract

The rise of cyberbullying in social media platforms involving toxic comments has escalated the need for effective ways to monitor and moderate online interactions. Existing solutions of automated toxicity detection systems, are based on a machine or deep learning algorithms. However, statistics-based solutions are generally prone to adversarial attacks that contain logic based modifications such as negation in phrases and sentences. In that regard, we present a set of formal reasoning-based methodologies that wrap around existing machine learning toxicity detection systems. Acting as both pre-processing and post-processing steps, our formal reasoning wrapper helps alleviating the negation attack problems and significantly improves the accuracy and efficacy of toxicity scoring. We evaluate different variations of our wrapper on multiple machine learning models against a negation adversarial dataset. Experimental results highlight the improvement of hybrid (formal reasoning and machine-learning) methods against various purely statistical solutions.

Not-in-Perspective: Towards Shielding Google's Perspective API Against Adversarial Negation Attacks

TL;DR

The paper tackles the vulnerability of Google's Perspective API to negation-based adversarial attacks in toxicity detection. It introduces a formal reasoning wrapper that operates as both pre-processing and post-processing around ML-based toxicity detectors, explicitly handling negation segments and exploring antonym substitution and paraphrase-based augmentation. The hybrid approaches M1.2-P and M1.5-P emerge as the strongest performers, delivering improved accuracy on Perspective-based negated test sets and reducing reported toxicity for many sentences, while Perspective remains a strong baseline. The work demonstrates that combining formal reasoning with statistical toxicity detectors yields more robust toxicity scoring, with practical implications for safer content moderation on social platforms.

Abstract

The rise of cyberbullying in social media platforms involving toxic comments has escalated the need for effective ways to monitor and moderate online interactions. Existing solutions of automated toxicity detection systems, are based on a machine or deep learning algorithms. However, statistics-based solutions are generally prone to adversarial attacks that contain logic based modifications such as negation in phrases and sentences. In that regard, we present a set of formal reasoning-based methodologies that wrap around existing machine learning toxicity detection systems. Acting as both pre-processing and post-processing steps, our formal reasoning wrapper helps alleviating the negation attack problems and significantly improves the accuracy and efficacy of toxicity scoring. We evaluate different variations of our wrapper on multiple machine learning models against a negation adversarial dataset. Experimental results highlight the improvement of hybrid (formal reasoning and machine-learning) methods against various purely statistical solutions.
Paper Structure (14 sections, 9 equations, 7 figures, 4 tables)

This paper contains 14 sections, 9 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Implementation
  4. Overall Methodology
  5. Reasoning and Synthesis
  6. Substitution and Exploration
  7. Evaluation
  8. Dataset and Models
  9. Results on Hosseini's Hosseini-RP3 Adversarial Set
  10. Results on Perspective-based Negated Testset
  11. Discussion
  12. Discussion on Results of Evaluation
  13. Discussion on Performance of Methods
  14. Conclusions & Future Work

Figures (7)

  • Figure 1: Dealing with negation.
  • Figure 2: Dealing with negation via replacement.
  • Figure 3: Substitution and Extrapolation's Workflow.
  • Figure 4: Percentage of improvement of toxicity in the Perspective-based negated public and private test sets using the Method M1.2-P.
  • Figure 5: Percentage of improvement of toxicity in the Perspective-based negated public and private test sets using the Method M1.5-P.
  • ...and 2 more figures