Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Models for the Detection of Hate, Abuse and Profanity

Christoph Tillmann, Aashka Trivedi, Bishwaranjan Bhattacharjee

TL;DR

The paper addresses the risk that web-trained LLMs produce hateful or profane outputs by developing efficient, encoder-based hap detectors with multilingual support. It introduces a small, fast 4-layer Piccolo architecture (ibm-hap-4l) alongside a larger bert-base hap classifier, both trained via fine-tuning and knowledge distillation, and leverages attention-heatmap visualizations for interpretability. The work demonstrates substantial latency and throughput improvements, HF Transformer compatibility, and practical deployment pathways, including data filtering, RL reward shaping, and generation control without retraining. It also extends to multilingual hap detection through scalable data collection, translation, and human validation, enabling broad language coverage for civil discourse across domains.

Abstract

Large Language Models (LLMs) are the cornerstone for many Natural Language Processing (NLP) tasks like sentiment analysis, document classification, named entity recognition, question answering, summarization, etc. LLMs are often trained on data which originates from the web. This data is prone to having content with Hate, Abuse and Profanity (HAP). For a detailed definition of HAP, please refer to the Appendix. Due to the LLMs being exposed to HAP content during training, the models learn it and may then generate hateful or profane content. For example, when the open-source RoBERTa model (specifically, the RoBERTA base model) from the HuggingFace (HF) Transformers library is prompted to replace the mask token in `I do not know that Persian people are that MASK` it returns the word `stupid` with the highest score. This is unacceptable in civil discourse.The detection of Hate, Abuse and Profanity in text is a vital component of creating civil and unbiased LLMs, which is needed not only for English, but for all languages. In this article, we briefly describe the creation of HAP detectors and various ways of using them to make models civil and acceptable in the output they generate.

Efficient Models for the Detection of Hate, Abuse and Profanity

TL;DR

The paper addresses the risk that web-trained LLMs produce hateful or profane outputs by developing efficient, encoder-based hap detectors with multilingual support. It introduces a small, fast 4-layer Piccolo architecture (ibm-hap-4l) alongside a larger bert-base hap classifier, both trained via fine-tuning and knowledge distillation, and leverages attention-heatmap visualizations for interpretability. The work demonstrates substantial latency and throughput improvements, HF Transformer compatibility, and practical deployment pathways, including data filtering, RL reward shaping, and generation control without retraining. It also extends to multilingual hap detection through scalable data collection, translation, and human validation, enabling broad language coverage for civil discourse across domains.

Abstract

Large Language Models (LLMs) are the cornerstone for many Natural Language Processing (NLP) tasks like sentiment analysis, document classification, named entity recognition, question answering, summarization, etc. LLMs are often trained on data which originates from the web. This data is prone to having content with Hate, Abuse and Profanity (HAP). For a detailed definition of HAP, please refer to the Appendix. Due to the LLMs being exposed to HAP content during training, the models learn it and may then generate hateful or profane content. For example, when the open-source RoBERTa model (specifically, the RoBERTA base model) from the HuggingFace (HF) Transformers library is prompted to replace the mask token in `I do not know that Persian people are that MASK` it returns the word `stupid` with the highest score. This is unacceptable in civil discourse.The detection of Hate, Abuse and Profanity in text is a vital component of creating civil and unbiased LLMs, which is needed not only for English, but for all languages. In this article, we briefly describe the creation of HAP detectors and various ways of using them to make models civil and acceptable in the output they generate.
Paper Structure (12 sections, 6 figures, 4 tables)

This paper contains 12 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Working of a hap Detector
  3. Model Architecture and Training
  4. Visualizing Sentence Toxicity using Attention Heatmaps
  5. Latency and Throughput
  6. Usage and Sample Code
  7. Applications and Use Cases
  8. Filtering Data for Language Model Training
  9. Reward Model for Reinforcement Learning in Model Alignment
  10. Controlling Outputs of Generative Models
  11. Multilingual hap Models
  12. hap Data Collection Effort

Figures (6)

  • Figure 1: hap output from Huggingface roberta Base model
  • Figure 2: Sample sentence with word-level self-attention. The sentence is tokenized with the WordPiece tokenizer devlin2019bert.
  • Figure 3: Generic Pipeline for the training and evaluation of language models. The hap mitigation step (highlighted in gray) can involve discarding documents with a large percentage of sentences having a high hap score, as given by a hap Classifier.
  • Figure 4: Use of the hap Model as a Reward Model in reinforcement learning to align the generative model to output less hap content.
  • Figure 5: Illustrative example for using hap Models for controlling generations. Here, a language model is prompted to fill the mask in "Arabs always smell like <mask>". Without any modification, the model would select the sequence that maximizes the Original Score ("Arabs always smell like sh*t"). Modifying the sampling process to choose the sequence that maximizes the New Score (the sum of the original score and the Non-hap Score), the generated sentence will be "Arabs always smell like roses".
  • ...and 1 more figures