ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Texts

TL;DR

ViGoEmotions presents a fine-grained Vietnamese emotion dataset with 20,664 comments annotated into 27 emotions plus Neutral, created via an LLM-assisted human-review pipeline inspired by GoEmotions. The work benchmarks eight Transformer architectures under three emoji/text preprocessing regimes, finding that emoji-preserving preprocessing with ViSoBERT yields the strongest results (test Macro F1 61.50% and Weighted F1 63.26%), while emoji-to-text helps some BERT-based baselines and lexical normalization with ViSoLex is competitive but not superior. Inter-annotator agreement analyses reveal variable but generally substantial agreement across emotions, and LLM-only labeling shows substantial but imperfect alignment with human judgments (exact match ~40.8%), underscoring the necessity of human verification. The study also analyzes emotion correlations, lexical patterns, and error modes, highlighting the importance of emoji behavior and domain adaptation for Vietnamese social-media emotion detection and providing a public benchmark for future research.

Abstract

Emotion classification plays a significant role in emotion prediction and harmful content detection. Recent advancements in NLP, particularly through large language models (LLMs), have greatly improved outcomes in this field. This study introduces ViGoEmotions -- a Vietnamese emotion corpus comprising 20,664 social media comments in which each comment is classified into 27 fine-grained distinct emotions. To evaluate the quality of the dataset and its impact on emotion classification, eight pre-trained Transformer-based models were evaluated under three preprocessing strategies: preserving original emojis with rule-based normalization, converting emojis into textual descriptions, and applying ViSoLex, a model-based lexical normalization system. Results show that converting emojis into text often improves the performance of several BERT-based baselines, while preserving emojis yields the best results for ViSoBERT and CafeBERT. In contrast, removing emojis generally leads to lower performance. ViSoBERT achieved the highest Macro F1-score of 61.50% and Weighted F1-score of 63.26%. Strong performance was also observed from CafeBERT and PhoBERT. These findings highlight that while the proposed corpus can support diverse architectures effectively, preprocessing strategies and annotation quality remain key factors influencing downstream performance.

Figures (7)

• Figure 1: Corpus building process.
• Figure 2: Average Cohen's Kappa Score Over Data Size
• Figure 3: Color indicates Kappa Score across emotion categories. Bars are sorted in descending order by number of samples.
• Figure 4: Labels Distribution in the ViGoEmotions
• Figure 5: Pearson Correlation Heatmap of Emotion Co-occurrences.