Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa

TL;DR

Japanese emotion classification remains challenging due to limited resources and severe class imbalance. The authors convert WRIME's intensity scores into eight independent binary labels and fine-tune four pretrained language models (BERT, RoBERTa, DeBERTa-base, DeBERTa-large), also comparing with two large language models. DeBERTa-v3-large delivers the strongest overall performance, achieving a mean accuracy of 0.860 and mean F1 of 0.662, and consistently outperforming LLMs on most emotions. The work provides a strong, downloadable tool for high-precision Japanese emotion detection and points to data augmentation and prompt engineering as fruitful directions for future improvements.

Abstract

Background Practical applications such as social media monitoring and customer-feedback analysis require accurate emotion detection for Japanese text, yet resource scarcity and class imbalance hinder model performance. Objective This study aims to build a high-accuracy model for predicting the presence or absence of eight Plutchik emotions in Japanese sentences. Methods Using the WRIME corpus, we transform reader-averaged intensity scores into binary labels and fine-tune four pre-trained language models (BERT, RoBERTa, DeBERTa-v3-base, DeBERTa-v3-large). For context, we also assess two large language models (TinySwallow-1.5B-Instruct and ChatGPT-4o). Accuracy and F1-score serve as evaluation metrics. Results DeBERTa-v3-large attains the best mean accuracy (0.860) and F1-score (0.662), outperforming all other models. It maintains robust F1 across both high-frequency emotions (e.g., Joy, Anticipation) and low-frequency emotions (e.g., Anger, Trust). The LLMs lag, with ChatGPT-4o and TinySwallow-1.5B-Instruct scoring 0.527 and 0.292 in mean F1, respectively. Conclusion The fine-tuned DeBERTa-v3-large model currently offers the most reliable solution for binary emotion classification in Japanese. We release this model as a pip-installable package (pip install deberta-emotion-predictor). Future work should augment data for rare emotions, reduce model size, and explore prompt engineering to improve LLM performance. This manuscript is under review for possible publication in New Generation Computing.