Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages

Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufino Cardenas, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ibrahim Lawan, Rooweither Mabuya, Rahmad Mahendra, Vukosi Marivate, Alexander Panchenko, Andrew Piper, Charles Henrique Porto Ferreira, Vitaly Protasov, Samuel Rutunda, Manish Shrivastava, Aura Cristina Udrea, Lilian Diana Awuor Wanzare, Sophie Wu, Florian Valentin Wunderlich, Hanif Muhammad Zhafran, Tianhui Zhang, Yi Zhou, Saif M. Mohammad

TL;DR

BRIGHTER tackles the scarcity of emotion recognition resources in under‑resourced languages by introducing a large, multi‑label, intensity‑annotated dataset across 28 languages and multiple domains. The paper details rigorous data collection, annotation, and reliability procedures, and evaluates monolingual and crosslingual transfer with both multilingual models and large language models, revealing persistent gaps in predicting perceived emotions and sensitivity to prompts. Key findings show transfer depends on language family and pretraining data, with English prompts often yielding better results for low‑resource languages, and that intensity prediction remains challenging. The work contributes a publicly available resource, annotation guidelines, and non‑aggregated labels, aiming to bridge the gap between language diversity and emotion‑recognition research while highlighting ethical considerations. $| ext{Emotion classes}| = 7$ (including neutral) and intensity ranges in $oxed{0 ext{ to } 3}$, with final labeling relying on thresholds such as $T = 0.5$ and $L_{ ext{final}}$ as defined in the aggregation rules.

Abstract

People worldwide use language in subtle and complex ways to express emotions. Although emotion recognition--an umbrella term for several NLP tasks--impacts various applications within NLP and beyond, most work in this area has focused on high-resource languages. This has led to significant disparities in research efforts and proposed solutions, particularly for under-resourced languages, which often lack high-quality annotated datasets. In this paper, we present BRIGHTER--a collection of multi-labeled, emotion-annotated datasets in 28 different languages and across several domains. BRIGHTER primarily covers low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers. We highlight the challenges related to the data collection and annotation processes, and then report experimental results for monolingual and crosslingual multi-label emotion identification, as well as emotion intensity recognition. We analyse the variability in performance across languages and text domains, both with and without the use of LLMs, and show that the BRIGHTER datasets represent a meaningful step towards addressing the gap in text-based emotion recognition.

BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages

TL;DR

BRIGHTER tackles the scarcity of emotion recognition resources in under‑resourced languages by introducing a large, multi‑label, intensity‑annotated dataset across 28 languages and multiple domains. The paper details rigorous data collection, annotation, and reliability procedures, and evaluates monolingual and crosslingual transfer with both multilingual models and large language models, revealing persistent gaps in predicting perceived emotions and sensitivity to prompts. Key findings show transfer depends on language family and pretraining data, with English prompts often yielding better results for low‑resource languages, and that intensity prediction remains challenging. The work contributes a publicly available resource, annotation guidelines, and non‑aggregated labels, aiming to bridge the gap between language diversity and emotion‑recognition research while highlighting ethical considerations. (including neutral) and intensity ranges in , with final labeling relying on thresholds such as and as defined in the aggregation rules.

Abstract

People worldwide use language in subtle and complex ways to express emotions. Although emotion recognition--an umbrella term for several NLP tasks--impacts various applications within NLP and beyond, most work in this area has focused on high-resource languages. This has led to significant disparities in research efforts and proposed solutions, particularly for under-resourced languages, which often lack high-quality annotated datasets. In this paper, we present BRIGHTER--a collection of multi-labeled, emotion-annotated datasets in 28 different languages and across several domains. BRIGHTER primarily covers low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers. We highlight the challenges related to the data collection and annotation processes, and then report experimental results for monolingual and crosslingual multi-label emotion identification, as well as emotion intensity recognition. We analyse the variability in performance across languages and text domains, both with and without the use of LLMs, and show that the BRIGHTER datasets represent a meaningful step towards addressing the gap in text-based emotion recognition.

Paper Structure

This paper contains 49 sections, 7 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. The Brighter Dataset Collection
  3. Data Sources
  4. Social media posts
  5. Personal narratives, talks, and speeches
  6. Literary texts
  7. News data
  8. Human-written and machine generated data
  9. Pre-processing and Quality Control
  10. Annotating Brighter
  11. Annotators' Reliability
  12. Determining the Final Labels
  13. Aggregating the emotion labels
  14. Aggregating the intensity labels
  15. Final Data Statistics
  16. ...and 34 more sections

Figures (8)

  • Figure 1: Languages included in Brighter and their language families.
  • Figure 2: Examples from the Brighter dataset collection in 6 different languages with their translations and intensity levels. Note that the instances can have one or more labels (e.g., disgust and surprise as shown in the figure).
  • Figure 3: Split-Half Class Match Percentage, SHCMP (%) values for the Brighter datasets across varying numbers of bins ($2$ to $10$). Higher values indicate better reliability scores. Note that ptmz and vwm have the same score as vwm instances were translated from ptmz and the translation was verified.
  • Figure 4: Emotion label distribution across the Brighter datasets. Each bar represents the number of labeled instances per emotion (i.e., anger, disgust, fear, joy, sadness, surprise, and neutral) and its percentage.
  • Figure 5: Ablation studies on the effect of prompt wording variation, few-shot examples, and pass@k predictions conducted on the English test set.
  • ...and 3 more figures