Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-Labeled Dataset for Indonesian Discourse: Examining Toxicity, Polarization, and Demographics Information

Lucky Susanto, Musa Wijanarko, Prasetia Pratama, Zilu Tang, Fariz Akyas, Traci Hong, Ika Idris, Alham Aji, Derry Wijaya

TL;DR

This work tackles the rising challenge of toxicity and political polarization in Indonesian online discourse by introducing a first-of-its-kind multi-label dataset that jointly annotates toxicity, polarization, and annotator demographics. The authors benchmark transformer-based models (notably IndoBERTweet) against large language models, showing that cross-task features and demographic information can improve both toxicity and polarization detection, with cross-task signals providing particularly strong gains for toxicity. They also reveal that polarization is more subjective and harder to detect than toxicity, and that demographic signals contribute meaningful gains when combined with cross-task information. The dataset enables analysis of the interaction between toxicity, polarization, and identities in a non-Western context, offering practical implications for moderation and policy in Indonesia while highlighting ethical considerations and the need for careful handling of annotator diversity and privacy.

Abstract

Polarization is defined as divisive opinions held by two or more groups on substantive issues. As the world's third-largest democracy, Indonesia faces growing concerns about the interplay between political polarization and online toxicity, which is often directed at vulnerable minority groups. Despite the importance of this issue, previous NLP research has not fully explored the relationship between toxicity and polarization. To bridge this gap, we present a novel multi-label Indonesian dataset that incorporates toxicity, polarization, and annotator demographic information. Benchmarking this dataset using BERT-base models and large language models (LLMs) shows that polarization information enhances toxicity classification, and vice versa. Furthermore, providing demographic information significantly improves the performance of polarization classification.

A Multi-Labeled Dataset for Indonesian Discourse: Examining Toxicity, Polarization, and Demographics Information

TL;DR

This work tackles the rising challenge of toxicity and political polarization in Indonesian online discourse by introducing a first-of-its-kind multi-label dataset that jointly annotates toxicity, polarization, and annotator demographics. The authors benchmark transformer-based models (notably IndoBERTweet) against large language models, showing that cross-task features and demographic information can improve both toxicity and polarization detection, with cross-task signals providing particularly strong gains for toxicity. They also reveal that polarization is more subjective and harder to detect than toxicity, and that demographic signals contribute meaningful gains when combined with cross-task information. The dataset enables analysis of the interaction between toxicity, polarization, and identities in a non-Western context, offering practical implications for moderation and policy in Indonesia while highlighting ethical considerations and the need for careful handling of annotator diversity and privacy.

Abstract

Polarization is defined as divisive opinions held by two or more groups on substantive issues. As the world's third-largest democracy, Indonesia faces growing concerns about the interplay between political polarization and online toxicity, which is often directed at vulnerable minority groups. Despite the importance of this issue, previous NLP research has not fully explored the relationship between toxicity and polarization. To bridge this gap, we present a novel multi-label Indonesian dataset that incorporates toxicity, polarization, and annotator demographic information. Benchmarking this dataset using BERT-base models and large language models (LLMs) shows that polarization information enhances toxicity classification, and vice versa. Furthermore, providing demographic information significantly improves the performance of polarization classification.

Paper Structure

This paper contains 60 sections, 1 equation, 2 figures, 27 tables.

Table of Contents

  1. Introduction
  2. Interplay Of Toxicity, Political Polarization, and Identity
  3. Toxicity and Polarization
  4. Non-Toxic Polarization
  5. How Identities Shape Discourse Dynamics
  6. Available Datasets
  7. Polarization Datasets
  8. Toxicity Datasets
  9. Toxicity and Polarization Dataset
  10. Dataset Creation
  11. Annotation Instrument
  12. Data Collection and Preprocessing
  13. Recruitment and Validation Metrics
  14. Annotation Process
  15. Dataset Properties
  16. ...and 45 more sections

Figures (2)

  • Figure 1: Samples of Toxic, Polarizing, alongside both Toxic and Polarizing texts.
  • Figure 2: This equation is used to calculate sample size $n$, where $z$ represents the Z-score associated with the confidence level, $p$ is the probability of a positive label, $e$ is the margin of error, and $N$ is the population size.