Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ContiGuard: A Framework for Continual Toxicity Detection Against Evolving Evasive Perturbations

Hankun Kang, Xin Miao, Jianhao Chen, Jintao Wen, Mayi Xu, Weiyu Zhang, Wenpeng Lu, Tieyun Qian

Abstract

Toxicity detection mitigates the dissemination of toxic content (e.g., hateful comments, posts, and messages within online social actions) to safeguard a healthy online social environment. However, malicious users persistently develop evasive perturbations to disguise toxic content and evade detectors. Traditional detectors or methods are static over time and are inadequate in addressing these evolving evasion tactics. Thus, continual learning emerges as a logical approach to dynamically update detection ability against evolving perturbations. Nevertheless, disparities across perturbations hinder the detector's continual learning on perturbed text. More importantly, perturbation-induced noises distort semantics to degrade comprehension and also impair critical feature learning to render detection sensitive to perturbations. These amplify the challenge of continual learning against evolving perturbations. In this work, we present ContiGuard, the first framework tailored for continual learning of the detector on time-evolving perturbed text (termed continual toxicity detection) to enable the detector to continually update capability and maintain sustained resilience against evolving perturbations. Specifically, to boost the comprehension, we present an LLM-powered semantic enriching strategy, where we dynamically incorporate possible meaning and toxicity-related clues excavated by LLM into the perturbed text to improve the comprehension. To mitigate non-critical features and amplify critical ones, we propose a discriminability-driven feature learning strategy, where we strengthen discriminative features while suppressing the less-discriminative ones to shape a robust classification boundary for detection...

ContiGuard: A Framework for Continual Toxicity Detection Against Evolving Evasive Perturbations

Abstract

Toxicity detection mitigates the dissemination of toxic content (e.g., hateful comments, posts, and messages within online social actions) to safeguard a healthy online social environment. However, malicious users persistently develop evasive perturbations to disguise toxic content and evade detectors. Traditional detectors or methods are static over time and are inadequate in addressing these evolving evasion tactics. Thus, continual learning emerges as a logical approach to dynamically update detection ability against evolving perturbations. Nevertheless, disparities across perturbations hinder the detector's continual learning on perturbed text. More importantly, perturbation-induced noises distort semantics to degrade comprehension and also impair critical feature learning to render detection sensitive to perturbations. These amplify the challenge of continual learning against evolving perturbations. In this work, we present ContiGuard, the first framework tailored for continual learning of the detector on time-evolving perturbed text (termed continual toxicity detection) to enable the detector to continually update capability and maintain sustained resilience against evolving perturbations. Specifically, to boost the comprehension, we present an LLM-powered semantic enriching strategy, where we dynamically incorporate possible meaning and toxicity-related clues excavated by LLM into the perturbed text to improve the comprehension. To mitigate non-critical features and amplify critical ones, we propose a discriminability-driven feature learning strategy, where we strengthen discriminative features while suppressing the less-discriminative ones to shape a robust classification boundary for detection...
Paper Structure (20 sections, 15 equations, 8 figures, 6 tables)

This paper contains 20 sections, 15 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. ContiGuard Framework
  5. LLM Powered Semantic Enriching Strategy
  6. Discriminability Driven Feature Learning Strategy
  7. Historical Capability Replay Strategy
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Analysis
  12. Conclusion
  13. Related Work
  14. Toxicity Detection
  15. Continual Learning
  16. ...and 5 more sections

Figures (8)

  • Figure 1: Illustration of our ContiGuard framework.
  • Figure 2: Results of statically fine-tuned detectors. Rows: types used for fine-tuning. Columns: types used for testing.
  • Figure 3: Results on different types of perturbed text at $T_9$. The numbers denote the best accuracy for each type.
  • Figure 4: Results of different memory sample numbers.
  • Figure 5: Retention rate of critical features, which denotes the proportion of historical critical features in current critical features. Each column shows the changes over time.
  • ...and 3 more figures