Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Emotion Detection on GoEmotions: Experimental Comparison of Classical Machine Learning, BiLSTM, and Transformer Models

Ani Harutyunyan, Sachin Kumar

TL;DR

Fine-grained emotion detection on the GoEmotions dataset is framed as a multi-label task with severe label imbalance. The authors compare three model families—TF-IDF logistic regression with Binary Relevance, a BiLSTM with Attention, and a BERT-based multi-label classifier—under a unified experimental setup using the official train/val/test splits and inverse-frequency weighting (with focal loss as a balance mechanism). They report that logistic regression achieves the highest Micro-F1 (0.51), while BERT delivers the strongest overall balance, with Macro-F1 0.49, Hamming Loss 0.036, and Subset Accuracy 0.36, highlighting the complementary roles of lexical cues and contextual representations for frequent versus rare emotions. These results illuminate practical trade-offs for imbalanced, multi-label emotion classification and suggest directions for threshold calibration, label-dependency modeling, and hybrid contextualization to further improve performance.

Abstract

Fine-grained emotion recognition is a challenging multi-label NLP task due to label overlap and class imbalance. In this work, we benchmark three modeling families on the GoEmotions dataset: a TF-IDF-based logistic regression system trained with binary relevance, a BiLSTM with attention, and a BERT model fine-tuned for multi-label classification. Experiments follow the official train/validation/test split, and imbalance is mitigated using inverse-frequency class weights. Across several metrics, namely Micro-F1, Macro-F1, Hamming Loss, and Subset Accuracy, we observe that logistic regression attains the highest Micro-F1 of 0.51, while BERT achieves the best overall balance surpassing the official paper's reported results, reaching Macro-F1 0.49, Hamming Loss 0.036, and Subset Accuracy 0.36. This suggests that frequent emotions often rely on surface lexical cues, whereas contextual representations improve performance on rarer emotions and more ambiguous examples.

Fine-Grained Emotion Detection on GoEmotions: Experimental Comparison of Classical Machine Learning, BiLSTM, and Transformer Models

TL;DR

Fine-grained emotion detection on the GoEmotions dataset is framed as a multi-label task with severe label imbalance. The authors compare three model families—TF-IDF logistic regression with Binary Relevance, a BiLSTM with Attention, and a BERT-based multi-label classifier—under a unified experimental setup using the official train/val/test splits and inverse-frequency weighting (with focal loss as a balance mechanism). They report that logistic regression achieves the highest Micro-F1 (0.51), while BERT delivers the strongest overall balance, with Macro-F1 0.49, Hamming Loss 0.036, and Subset Accuracy 0.36, highlighting the complementary roles of lexical cues and contextual representations for frequent versus rare emotions. These results illuminate practical trade-offs for imbalanced, multi-label emotion classification and suggest directions for threshold calibration, label-dependency modeling, and hybrid contextualization to further improve performance.

Abstract

Fine-grained emotion recognition is a challenging multi-label NLP task due to label overlap and class imbalance. In this work, we benchmark three modeling families on the GoEmotions dataset: a TF-IDF-based logistic regression system trained with binary relevance, a BiLSTM with attention, and a BERT model fine-tuned for multi-label classification. Experiments follow the official train/validation/test split, and imbalance is mitigated using inverse-frequency class weights. Across several metrics, namely Micro-F1, Macro-F1, Hamming Loss, and Subset Accuracy, we observe that logistic regression attains the highest Micro-F1 of 0.51, while BERT achieves the best overall balance surpassing the official paper's reported results, reaching Macro-F1 0.49, Hamming Loss 0.036, and Subset Accuracy 0.36. This suggests that frequent emotions often rely on surface lexical cues, whereas contextual representations improve performance on rarer emotions and more ambiguous examples.
Paper Structure (8 sections, 16 equations, 5 figures, 6 tables)

This paper contains 8 sections, 16 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Methodology
  4. Dataset Description
  5. Proposed Framework
  6. Evaluation Measures/Metrics
  7. Experiments, Results and Discussion
  8. Conclusion, Limitations and Future Directions

Figures (5)

  • Figure 1: Emotion Label Distribution: The chart presents the distribution of emotion labels across the dataset. The most frequent emotion after neutral is is Admiration, followed by Approval and Gratitude.
  • Figure 2: Wordclouds for the Emotion Categories - Caring, Grief, Excitement. The token "NAME" represents anonymized user mentions in the original comments.
  • Figure 3: Pipeline Stages: We propose a 4 stage pipeline, which is adapted to the dataset's unique nature and addresses its limitations
  • Figure 4: F1 Score comparison across all emotion categories
  • Figure 5: Precision-Recall Trade off analysis for each model