Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentation

TL;DR

Three key approaches for attaining smaller and more efficient NLP models are employed, training a Transformer-based neural network to detect offensive language, employing data augmentation and knowledge distillation techniques to increase performance, and incorporating multi-task learning with knowledge distillation and teacher annealing using diverse datasets to enhance efficiency.

Abstract

This paper highlights the significance of natural language processing (NLP) within artificial intelligence, underscoring its pivotal role in comprehending and modeling human language. Recent advancements in NLP, particularly in conversational bots, have garnered substantial attention and adoption among developers. This paper explores advanced methodologies for attaining smaller and more efficient NLP models. Specifically, we employ three key approaches: (1) training a Transformer-based neural network to detect offensive language, (2) employing data augmentation and knowledge distillation techniques to increase performance, and (3) incorporating multi-task learning with knowledge distillation and teacher annealing using diverse datasets to enhance efficiency. The culmination of these methods has yielded demonstrably improved outcomes.

Figures (1)

• Figure 1: Multi-task learning with knowledge distillation and teacher annealing.