Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainability-Based Token Replacement on LLM-Generated Text

Hadi Mohammadi, Anastasia Giachanou, Daniel L. Oberski, Ayoub Bagheri

TL;DR

The paper investigates whether explainability methods can be weaponized to hide AI-generated text while maintaining detection through a robust ensemble. It introduces SHAP/LIME-guided token replacement strategies and four rewriting methods, assessing their effect on detection and textual fidelity across English and Dutch, in multiple domains. A novel frozen-plus-fresh BERT ensemble is shown to be more robust than single-model detectors, with cross-domain and cross-language resilience despite targeted rewrites. The study highlights a dynamic interplay between XAI-driven evasion and ensemble defenses, emphasizing the need for multi-model governance, watermarking integration, and ongoing adaptation as AI text generation evolves. Practically, it informs detector design and policy for resilient, transparent AI-generated content moderation across languages and genres.

Abstract

Generative models, especially large language models (LLMs), have shown remarkable progress in producing text that appears human-like. However, they often exhibit patterns that make their output easier to detect than text written by humans. In this paper, we investigate how explainable AI (XAI) methods can be used to reduce the detectability of AI-generated text (AIGT) while also introducing a robust ensemble-based detection approach. We begin by training an ensemble classifier to distinguish AIGT from human-written text, then apply SHAP and LIME to identify tokens that most strongly influence its predictions. We propose four explainability-based token replacement strategies to modify these influential tokens. Our findings show that these token replacement approaches can significantly diminish a single classifier's ability to detect AIGT. However, our ensemble classifier maintains strong performance across multiple languages and domains, showing that a multi-model approach can mitigate the impact of token-level manipulations. These results show that XAI methods can make AIGT harder to detect by focusing on the most influential tokens. At the same time, they highlight the need for robust, ensemble-based detection strategies that can adapt to evolving approaches for hiding AIGT.

Explainability-Based Token Replacement on LLM-Generated Text

TL;DR

The paper investigates whether explainability methods can be weaponized to hide AI-generated text while maintaining detection through a robust ensemble. It introduces SHAP/LIME-guided token replacement strategies and four rewriting methods, assessing their effect on detection and textual fidelity across English and Dutch, in multiple domains. A novel frozen-plus-fresh BERT ensemble is shown to be more robust than single-model detectors, with cross-domain and cross-language resilience despite targeted rewrites. The study highlights a dynamic interplay between XAI-driven evasion and ensemble defenses, emphasizing the need for multi-model governance, watermarking integration, and ongoing adaptation as AI text generation evolves. Practically, it informs detector design and policy for resilient, transparent AI-generated content moderation across languages and genres.

Abstract

Generative models, especially large language models (LLMs), have shown remarkable progress in producing text that appears human-like. However, they often exhibit patterns that make their output easier to detect than text written by humans. In this paper, we investigate how explainable AI (XAI) methods can be used to reduce the detectability of AI-generated text (AIGT) while also introducing a robust ensemble-based detection approach. We begin by training an ensemble classifier to distinguish AIGT from human-written text, then apply SHAP and LIME to identify tokens that most strongly influence its predictions. We propose four explainability-based token replacement strategies to modify these influential tokens. Our findings show that these token replacement approaches can significantly diminish a single classifier's ability to detect AIGT. However, our ensemble classifier maintains strong performance across multiple languages and domains, showing that a multi-model approach can mitigate the impact of token-level manipulations. These results show that XAI methods can make AIGT harder to detect by focusing on the most influential tokens. At the same time, they highlight the need for robust, ensemble-based detection strategies that can adapt to evolving approaches for hiding AIGT.

Paper Structure

This paper contains 36 sections, 1 equation, 3 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Methodology
  5. Baselines Models.
  6. Ensemble Model.
  7. Experimental Setup
  8. Computational Resources.
  9. Explainability-driven detection
  10. Token Replacement Strategies
  11. Evaluation Metrics
  12. Experimental Analysis and Results
  13. General Trends.
  14. Comparison of Token-Selection Approaches.
  15. Influence of Rewriting Methods.
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Combined BERT Ensemble Structure
  • Figure 1: The architecture of the proposed ensemble model. Outputs of three frozen BERT models (left) and three fresh BERT models (right) are concatenated and passed through a dense layer followed by a sigmoid activation (and threshold at 0.5) for final binary predictions.
  • Figure 2: Overlaps among flipped AI samples under different explainability methods and rewriting strategies. Each bar corresponds to the number of samples simultaneously flipped by one or more models.