Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models

Sargam Yadav, Abhishek Kaushik, Kevin McDaid

TL;DR

This paper tackles hate speech detection in code-mixed Hinglish under data scarcity by evaluating zero-shot, one-shot, and few-shot transfer-learning approaches using large language models. The authors assemble a weakly annotated dataset of 100 Hinglish YouTube comments (MGY/NOT) with nine fine-grained misogyny labels and test BART-large for zero-shot classification, SetFit with MPNet for one-/few-shot, and ChatGPT-3 prompting for one-shot labeling. Results show zero-shot achieves up to 54% accuracy, few-shot about 51%, while one-shot performs poorly (~33.7%), with multi-label metrics remaining modest; ChatGPT-3 prompts demonstrate promise for capturing some fine-grained categories. The study demonstrates the feasibility of LLM-based transfer learning to assist annotation in low-resource code-mixed contexts, while highlighting limitations such as dataset size, single annotator bias, and the need for larger, multilingual evaluations to validate the approach.

Abstract

The advent of Large Language Models (LLMs) has advanced the benchmark in various Natural Language Processing (NLP) tasks. However, large amounts of labelled training data are required to train LLMs. Furthermore, data annotation and training are computationally expensive and time-consuming. Zero and few-shot learning have recently emerged as viable options for labelling data using large pre-trained models. Hate speech detection in mix-code low-resource languages is an active problem area where the use of LLMs has proven beneficial. In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish. Weak annotation was applied due to the labor-intensive annotation process. Zero-shot learning, one-shot learning, and few-shot learning and prompting approaches have then been applied to assign labels to the comments and compare them to human-assigned labels. Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results

Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models

TL;DR

This paper tackles hate speech detection in code-mixed Hinglish under data scarcity by evaluating zero-shot, one-shot, and few-shot transfer-learning approaches using large language models. The authors assemble a weakly annotated dataset of 100 Hinglish YouTube comments (MGY/NOT) with nine fine-grained misogyny labels and test BART-large for zero-shot classification, SetFit with MPNet for one-/few-shot, and ChatGPT-3 prompting for one-shot labeling. Results show zero-shot achieves up to 54% accuracy, few-shot about 51%, while one-shot performs poorly (~33.7%), with multi-label metrics remaining modest; ChatGPT-3 prompts demonstrate promise for capturing some fine-grained categories. The study demonstrates the feasibility of LLM-based transfer learning to assist annotation in low-resource code-mixed contexts, while highlighting limitations such as dataset size, single annotator bias, and the need for larger, multilingual evaluations to validate the approach.

Abstract

The advent of Large Language Models (LLMs) has advanced the benchmark in various Natural Language Processing (NLP) tasks. However, large amounts of labelled training data are required to train LLMs. Furthermore, data annotation and training are computationally expensive and time-consuming. Zero and few-shot learning have recently emerged as viable options for labelling data using large pre-trained models. Hate speech detection in mix-code low-resource languages is an active problem area where the use of LLMs has proven beneficial. In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish. Weak annotation was applied due to the labor-intensive annotation process. Zero-shot learning, one-shot learning, and few-shot learning and prompting approaches have then been applied to assign labels to the comments and compare them to human-assigned labels. Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results
Paper Structure (17 sections, 5 figures, 3 tables)

This paper contains 17 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Literature Review
  4. Hate Speech Classifiers and Code-mixed Languages
  5. Zero-Shot, One-shot, and Few-Shot Learning
  6. Methodology
  7. Dataset
  8. Experiment
  9. Zero-shot Learning
  10. Few-shot Learning
  11. One-shot prompting through ChatGPT-3
  12. Results and Analysis
  13. Zero-Shot, One-Shot, and Few-shot Classification
  14. One-shot Prompting
  15. Discussion
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Example of a sentence from the dataset belonging to class MGY
  • Figure 2: Methodology of the proposed study
  • Figure 3: Zero-Shot Classification
  • Figure 4: Confusion Matrix for Classifiers
  • Figure 5: Bar Graph for Binary Classification