Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Efficient and Explainable Hate Speech Detection via Model Distillation

Paloma Piot, Javier Parapar

TL;DR

This paper addresses the cost and opacity of large language models in hate speech detection by proposing a knowledge-distillation pipeline that uses Few-Shot Chain-of-Thought rationales from a 70B teacher to train a smaller 8B student in a multi-task setting for both classification and explanation. The distilled model achieves competitive or superior classification performance while providing high-quality explanations, with substantial gains in efficiency and reduced resource usage. Human evaluation confirms the quality of the generated rationales, supporting trust and transparency in moderation decisions. The work demonstrates practical implications for deploying explainable hate speech detection in real-world platforms under regulatory demands while enabling scalable, responsible moderation at lower computational cost.

Abstract

Automatic detection of hate and abusive language is essential to combat its online spread. Moreover, recognising and explaining hate speech serves to educate people about its negative effects. However, most current detection models operate as black boxes, lacking interpretability and explainability. In this context, Large Language Models (LLMs) have proven effective for hate speech detection and to promote interpretability. Nevertheless, they are computationally costly to run. In this work, we propose distilling big language models by using Chain-of-Thought to extract explanations that support the hate speech classification task. Having small language models for these tasks will contribute to their use in operational settings. In this paper, we demonstrate that distilled models deliver explanations of the same quality as larger models while surpassing them in classification performance. This dual capability, classifying and explaining, advances hate speech detection making it more affordable, understandable and actionable.

Towards Efficient and Explainable Hate Speech Detection via Model Distillation

TL;DR

This paper addresses the cost and opacity of large language models in hate speech detection by proposing a knowledge-distillation pipeline that uses Few-Shot Chain-of-Thought rationales from a 70B teacher to train a smaller 8B student in a multi-task setting for both classification and explanation. The distilled model achieves competitive or superior classification performance while providing high-quality explanations, with substantial gains in efficiency and reduced resource usage. Human evaluation confirms the quality of the generated rationales, supporting trust and transparency in moderation decisions. The work demonstrates practical implications for deploying explainable hate speech detection in real-world platforms under regulatory demands while enabling scalable, responsible moderation at lower computational cost.

Abstract

Automatic detection of hate and abusive language is essential to combat its online spread. Moreover, recognising and explaining hate speech serves to educate people about its negative effects. However, most current detection models operate as black boxes, lacking interpretability and explainability. In this context, Large Language Models (LLMs) have proven effective for hate speech detection and to promote interpretability. Nevertheless, they are computationally costly to run. In this work, we propose distilling big language models by using Chain-of-Thought to extract explanations that support the hate speech classification task. Having small language models for these tasks will contribute to their use in operational settings. In this paper, we demonstrate that distilled models deliver explanations of the same quality as larger models while surpassing them in classification performance. This dual capability, classifying and explaining, advances hate speech detection making it more affordable, understandable and actionable.

Paper Structure

This paper contains 20 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposal
  4. Knowledge Distillation
  5. Extract Rationales
  6. Train Small Model with Rationales
  7. Experiments
  8. Datasets
  9. Chosen LLMs
  10. Experimental Settings
  11. Experiment 1: Classification Task
  12. Experiment 2: Explanation Task
  13. Hate Speech Definition
  14. Annotation process
  15. Annotators
  16. ...and 5 more sections

Figures (3)

  • Figure 1: X's Community Notes feature for adding post context.
  • Figure 2: LLM Knowledge distillation with CoT overview.
  • Figure 3: Overview of our proposed approach for explaining and detecting hate speech: First, we extract rationales from an LLM using Few-Shot CoT. We then use these rationales, along with the labels, to train a small model within a multi-task learning framework. This enables the small model to explain and detect hate speech effectively.