Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Characterising Toxicity in Generative Large Language Models

Zhiyao Zhang, Yazan Mash'Al, Yuhan Wu

TL;DR

This work investigates how generative large language models produce toxic outputs when prompted and what linguistic features drive such toxicity. It combines a toxic/benign prompt dataset (DecodingTrust) with seven diverse LLMs, automated Perspective API toxicity scoring, and attribution-based lexical analysis (Captum) plus syntactic analysis (POS tagging, dependency parsing) to uncover patterns. Key findings show that toxic prompts reliably elicit toxic responses in base models (with $P_ ext{tox}$ often very high), while instruction-tuned models mitigate toxicity but do not eliminate it; lexical triggers are dominated by content-specific nouns, and syntactic ROOT structures (short imperative clauses) amplify toxicity. The study informs mitigation strategies including stronger input moderation, refined fine-tuning, and context-aware evaluation, while acknowledging limitations of automated toxicity measures and the need for human-in-the-loop assessment to ensure robust, socially responsible LLM behavior.

Abstract

In recent years, the advent of the attention mechanism has significantly advanced the field of natural language processing (NLP), revolutionizing text processing and text generation. This has come about through transformer-based decoder-only architectures, which have become ubiquitous in NLP due to their impressive text processing and generation capabilities. Despite these breakthroughs, language models (LMs) remain susceptible to generating undesired outputs: inappropriate, offensive, or otherwise harmful responses. We will collectively refer to these as ``toxic'' outputs. Although methods like reinforcement learning from human feedback (RLHF) have been developed to align model outputs with human values, these safeguards can often be circumvented through carefully crafted prompts. Therefore, this paper examines the extent to which LLMs generate toxic content when prompted, as well as the linguistic factors -- both lexical and syntactic -- that influence the production of such outputs in generative models.

Characterising Toxicity in Generative Large Language Models

TL;DR

This work investigates how generative large language models produce toxic outputs when prompted and what linguistic features drive such toxicity. It combines a toxic/benign prompt dataset (DecodingTrust) with seven diverse LLMs, automated Perspective API toxicity scoring, and attribution-based lexical analysis (Captum) plus syntactic analysis (POS tagging, dependency parsing) to uncover patterns. Key findings show that toxic prompts reliably elicit toxic responses in base models (with often very high), while instruction-tuned models mitigate toxicity but do not eliminate it; lexical triggers are dominated by content-specific nouns, and syntactic ROOT structures (short imperative clauses) amplify toxicity. The study informs mitigation strategies including stronger input moderation, refined fine-tuning, and context-aware evaluation, while acknowledging limitations of automated toxicity measures and the need for human-in-the-loop assessment to ensure robust, socially responsible LLM behavior.

Abstract

In recent years, the advent of the attention mechanism has significantly advanced the field of natural language processing (NLP), revolutionizing text processing and text generation. This has come about through transformer-based decoder-only architectures, which have become ubiquitous in NLP due to their impressive text processing and generation capabilities. Despite these breakthroughs, language models (LMs) remain susceptible to generating undesired outputs: inappropriate, offensive, or otherwise harmful responses. We will collectively refer to these as ``toxic'' outputs. Although methods like reinforcement learning from human feedback (RLHF) have been developed to align model outputs with human values, these safeguards can often be circumvented through carefully crafted prompts. Therefore, this paper examines the extent to which LLMs generate toxic content when prompted, as well as the linguistic factors -- both lexical and syntactic -- that influence the production of such outputs in generative models.
Paper Structure (31 sections, 4 figures, 7 tables)

This paper contains 31 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Methodology
  4. Input Dataset
  5. Selected Large Language Models
  6. Quantitative Analysis Framework
  7. Qualitative Analysis Framework
  8. Results
  9. Research Question 1
  10. Evaluating the toxicity of generated outputs
  11. Comparing the levels of toxicity categories across language models
  12. Qualitative analysis of toxicity across different LLMs
  13. Conclusion of Research Question 1
  14. Research Question 2
  15. Choosing a toxic subset
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Comparison of model behavior with toxic and non-toxic inputs.
  • Figure 2: Comparison of high toxicity thresholds (score $\geq 0.8$) with toxic and non-toxic inputs.
  • Figure 3: POS Tag Distribution for High-Attribution Tokens
  • Figure 4: Dependency Label Distribution for High-Attribution Tokens