Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"Define Your Terms" : Enhancing Efficient Offensive Speech Classification with Definition

Huy Nghiem, Umang Gupta, Fred Morstatter

TL;DR

Problem: cross-domain offensive speech detection with limited labeled data faces semantic diversity and domain shift. Method: JE_ProtoNet, a joint embedding Prototypical Network that uses label definitions via attention to align inputs with label semantics, trained with meta-learning across 10 domains and evaluated on 4 held-out domains. Contributions: compilation of 14 datasets (~82k training samples across 10 domains), demonstration that label-definition-aware joint embeddings yield robust few-shot performance (achieving at least $75\%$ of the maximal $F1$-score with $<10\%$ of data), and extensive ablations showing the benefit of definition-aware signaling. Significance: enhances data efficiency for offensive-speech detection and provides a framework likely applicable to other label-rich NLP tasks such as sentiment or stance detection.

Abstract

The propagation of offensive content through social media channels has garnered attention of the research community. Multiple works have proposed various semantically related yet subtle distinct categories of offensive speech. In this work, we explore meta-earning approaches to leverage the diversity of offensive speech corpora to enhance their reliable and efficient detection. We propose a joint embedding architecture that incorporates the input's label and definition for classification via Prototypical Network. Our model achieves at least 75% of the maximal F1-score while using less than 10% of the available training data across 4 datasets. Our experimental findings also provide a case study of training strategies valuable to combat resource scarcity.

"Define Your Terms" : Enhancing Efficient Offensive Speech Classification with Definition

TL;DR

Problem: cross-domain offensive speech detection with limited labeled data faces semantic diversity and domain shift. Method: JE_ProtoNet, a joint embedding Prototypical Network that uses label definitions via attention to align inputs with label semantics, trained with meta-learning across 10 domains and evaluated on 4 held-out domains. Contributions: compilation of 14 datasets (~82k training samples across 10 domains), demonstration that label-definition-aware joint embeddings yield robust few-shot performance (achieving at least of the maximal -score with of data), and extensive ablations showing the benefit of definition-aware signaling. Significance: enhances data efficiency for offensive-speech detection and provides a framework likely applicable to other label-rich NLP tasks such as sentiment or stance detection.

Abstract

The propagation of offensive content through social media channels has garnered attention of the research community. Multiple works have proposed various semantically related yet subtle distinct categories of offensive speech. In this work, we explore meta-earning approaches to leverage the diversity of offensive speech corpora to enhance their reliable and efficient detection. We propose a joint embedding architecture that incorporates the input's label and definition for classification via Prototypical Network. Our model achieves at least 75% of the maximal F1-score while using less than 10% of the available training data across 4 datasets. Our experimental findings also provide a case study of training strategies valuable to combat resource scarcity.
Paper Structure (19 sections, 3 equations, 3 figures, 7 tables)

This paper contains 19 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Annotation with Instructions
  4. Cross-Dataset Transference
  5. Label-Aware Classification
  6. Data Collection and Processing
  7. Experimental Setup
  8. Baselines
  9. Meta-Learning Settings
  10. Training with Label Information
  11. Results
  12. For Baseline Settings
  13. For Meta-Learning Settings
  14. Without Label
  15. With Label
  16. ...and 4 more sections

Figures (3)

  • Figure 1: General architecture of JE_ProtoNet. Hidden states of text input and its corresponding label and definition are obtained from RoBERTa, and then passed as Query, Value, and Key (color coded for traceability) to the Multiheaded Attention module.
  • Figure 2: Illustration of Macro F1-scores of models for various K-shot settings. Vertical bars denote standard deviation of results over 5 seeds. HateBERT is ommited to simplify comparison.
  • Figure 3: Mean macro F-1 scores of various models on 4 test sets at different K-shot settings, with error bars representing standard deviation over 5 seeds . For each K, the first bar shows the best performer among the Baseline models, the second bar shows the best among the models Without Label, and the third among models With Label. The rest includes all applicable Joint-Embedding models. R_Bi: RoBERTa_binary, R_Re: RoBERTa_retrained, R_Un: RoBERTa_untrained, PN: ProtoNet, PN_F: ProtoNET_Full, PN_T: ProtoNet_Token, PN_L: ProtoNet_Label, PM: ProtoMAML, JE_PN: JE_ProtoNet, JE_PN_U: JE_ProtoNet_Untrained, JE_PN_C: JE_ProtoNet_CLS.