Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploration and Evaluation of Bias in Cyberbullying Detection with Machine Learning

Andrew Root, Liam Jakubowski, Mounika Vanamala

TL;DR

The paper investigates how varying definitions, labeling schemes, and data collection methods bias cyberbullying detection models and degrade generalization to unseen data. It analyzes three Twitter-based datasets, standardizes labels to a binary cyberbullying/not category, and employs six-fold stratified cross-validation alongside cross-dataset evaluations, introducing Dynamic Query Expansion as a dataset augmentation method. The key finding is a substantial average drop of Macro F1 by $0.222$ (and Weighted F1 by $0.253$) when models are tested on different datasets, underscoring the fragility of cross-domain performance and the necessity of careful dataset curation and cross-dataset testing. The work highlights the dangers of lexicon-based data collection and automated expansion, calling for bias-aware data practices to ensure real-world applicability of cyberbullying detection systems.

Abstract

It is well known that the usefulness of a machine learning model is due to its ability to generalize to unseen data. This study uses three popular cyberbullying datasets to explore the effects of data, how it's collected, and how it's labeled, on the resulting machine learning models. The bias introduced from differing definitions of cyberbullying and from data collection is discussed in detail. An emphasis is made on the impact of dataset expansion methods, which utilize current data points to fetch and label new ones. Furthermore, explicit testing is performed to evaluate the ability of a model to generalize to unseen datasets through cross-dataset evaluation. As hypothesized, the models have a significant drop in the Macro F1 Score, with an average drop of 0.222. As such, this study effectively highlights the importance of dataset curation and cross-dataset testing for creating models with real-world applicability. The experiments and other code can be found at https://github.com/rootdrew27/cyberbullying-ml.

Exploration and Evaluation of Bias in Cyberbullying Detection with Machine Learning

TL;DR

The paper investigates how varying definitions, labeling schemes, and data collection methods bias cyberbullying detection models and degrade generalization to unseen data. It analyzes three Twitter-based datasets, standardizes labels to a binary cyberbullying/not category, and employs six-fold stratified cross-validation alongside cross-dataset evaluations, introducing Dynamic Query Expansion as a dataset augmentation method. The key finding is a substantial average drop of Macro F1 by (and Weighted F1 by ) when models are tested on different datasets, underscoring the fragility of cross-domain performance and the necessity of careful dataset curation and cross-dataset testing. The work highlights the dangers of lexicon-based data collection and automated expansion, calling for bias-aware data practices to ensure real-world applicability of cyberbullying detection systems.

Abstract

It is well known that the usefulness of a machine learning model is due to its ability to generalize to unseen data. This study uses three popular cyberbullying datasets to explore the effects of data, how it's collected, and how it's labeled, on the resulting machine learning models. The bias introduced from differing definitions of cyberbullying and from data collection is discussed in detail. An emphasis is made on the impact of dataset expansion methods, which utilize current data points to fetch and label new ones. Furthermore, explicit testing is performed to evaluate the ability of a model to generalize to unseen datasets through cross-dataset evaluation. As hypothesized, the models have a significant drop in the Macro F1 Score, with an average drop of 0.222. As such, this study effectively highlights the importance of dataset curation and cross-dataset testing for creating models with real-world applicability. The experiments and other code can be found at https://github.com/rootdrew27/cyberbullying-ml.

Paper Structure

This paper contains 16 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Datasets
  4. Dataset 1
  5. Dataset 2
  6. Dataset 3
  7. Methods
  8. Theory
  9. Preprocessing
  10. Cross Validation
  11. Cross-Dataset Evaluation
  12. Results
  13. Cross Validation Results
  14. Cross-Dataset Evaluation Results
  15. Discussion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: The Experiment Methodology. The process is performed for each Dataset x and for each of the remaining datasets y such that six total experiments are performed. Note that the Cleaning, Cross Validation. and Cross-Dataset Evaluation only vary with regard to the inputs depicted.
  • Figure 2: Cross Validation Results
  • Figure 3: Cross-Dataset Evaluation Results. The red dashed line indicates the average across all experiments.