Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

Luan Thanh Nguyen

TL;DR

ViHateT5 introduces a domain-specific, unified text-to-text transformer for Vietnamese hate speech detection, trained on the VOZ-HSD corpus of over 10 million social-media comments. By framing three HSD tasks (hate speech detection, toxic speech detection, and hate spans detection) within a single T5 model and employing task-prefixes and span-markers, ViHateT5 achieves state-of-the-art results across Vietnamese HSD benchmarks. The study demonstrates the importance of domain-specific pre-training and label distribution, and provides open access to VOZ-HSD, pre-trained checkpoints, and code to foster further research. This work reduces system fragmentation and enhances practical content moderation for Vietnamese online discourse.

Abstract

Recent advancements in hate speech detection (HSD) in Vietnamese have made significant progress, primarily attributed to the emergence of transformer-based pre-trained language models, particularly those built on the BERT architecture. However, the necessity for specialized fine-tuned models has resulted in the complexity and fragmentation of developing a multitasking HSD system. Moreover, most current methodologies focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia, which may not accurately capture human behavior on online platforms. In this research, we introduce ViHateT5, a T5-based model pre-trained on our proposed large-scale domain-specific dataset named VOZ-HSD. By harnessing the power of a text-to-text architecture, ViHateT5 can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. Our experiments also underscore the significance of label distribution in pre-training data on model efficacy. We provide our experimental materials for research purposes, including the VOZ-HSD dataset, pre-trained checkpoint, the unified HSD-multitask ViHateT5 model, and related source code on GitHub publicly.

ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

TL;DR

ViHateT5 introduces a domain-specific, unified text-to-text transformer for Vietnamese hate speech detection, trained on the VOZ-HSD corpus of over 10 million social-media comments. By framing three HSD tasks (hate speech detection, toxic speech detection, and hate spans detection) within a single T5 model and employing task-prefixes and span-markers, ViHateT5 achieves state-of-the-art results across Vietnamese HSD benchmarks. The study demonstrates the importance of domain-specific pre-training and label distribution, and provides open access to VOZ-HSD, pre-trained checkpoints, and code to foster further research. This work reduces system fragmentation and enhances practical content moderation for Vietnamese online discourse.

Abstract

Recent advancements in hate speech detection (HSD) in Vietnamese have made significant progress, primarily attributed to the emergence of transformer-based pre-trained language models, particularly those built on the BERT architecture. However, the necessity for specialized fine-tuned models has resulted in the complexity and fragmentation of developing a multitasking HSD system. Moreover, most current methodologies focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia, which may not accurately capture human behavior on online platforms. In this research, we introduce ViHateT5, a T5-based model pre-trained on our proposed large-scale domain-specific dataset named VOZ-HSD. By harnessing the power of a text-to-text architecture, ViHateT5 can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. Our experiments also underscore the significance of label distribution in pre-training data on model efficacy. We provide our experimental materials for research purposes, including the VOZ-HSD dataset, pre-trained checkpoint, the unified HSD-multitask ViHateT5 model, and related source code on GitHub publicly.
Paper Structure (26 sections, 3 figures, 11 tables, 2 algorithms)

This paper contains 26 sections, 3 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. ViHateT5
  4. Automated Pre-training Data Creation
  5. Model Pre-training
  6. Model Fine-tuning
  7. Experiments and Results
  8. Data
  9. Model Setup
  10. Baseline
  11. Multilingual Pre-trained Models
  12. Monolingual Pre-trained Models
  13. Evaluation
  14. Experimental Results
  15. Discussion
  16. ...and 11 more sections

Figures (3)

  • Figure 1: An overview of the unified HSD-multitask ViHateT5 model incorporating various prefix tasks tailored for hate speech detection in Vietnamese.
  • Figure 2: The process of creating VOZ-HSD by the automated data labeling approach.
  • Figure 3: The word cloud of VOZ-HSD dataset.