Automatic Classifiers Underdetect Emotions Expressed by Men

TL;DR

Investigating gender biases in emotion detection across 414 combinations of models and emotion-related classes finds that across different types of automatic classifiers and various underlying emotions, error rates are consistently higher for texts authored by men compared to those authored by women.

Abstract

The widespread adoption of automatic sentiment and emotion classifiers makes it important to ensure that these tools perform reliably across different populations. Yet their reliability is typically assessed using benchmarks that rely on third-party annotators rather than the individuals experiencing the emotions themselves, potentially concealing systematic biases. In this paper, we use a unique, large-scale dataset of more than one million self-annotated posts and a pre-registered research design to investigate gender biases in emotion detection across 414 combinations of models and emotion-related classes. We find that across different types of automatic classifiers and various underlying emotions, error rates are consistently higher for texts authored by men compared to those authored by women. We quantify how this bias could affect results in downstream applications and show that current machine learning tools, including large language models, should be applied with caution when the gender composition of a sample is not known or variable. Our findings demonstrate that sentiment analysis is not yet a solved problem, especially in ensuring equitable model behaviour across demographic groups.

Figures (4)

• Figure 1: Error rates for men are consistently higher than those for women across all model types for both positive and negative emotions. The men bias is quantified as the relative increase in error rate, i.e. $(ER_{\text{men}} - ER_{\text{women}}) / ER_{\text{women}}$. Bootstrap resampling with $10,000$ iterations was used to compute 95% confidence intervals. Note that while it might appear that the salience error for negative posts is significantly higher for women with the NRC-lexicon-based approach (negative value for the men bias), the corresponding p-value does not reach the significance threshold specified in the pre-registration.
• Figure 2: Error rates for men are consistently higher than those for women across both types of error and all levels of analysis: individual mood tags, emotion classes, and sentiment. The difference is more pronounced for salience error; however, only a limited number of models return neutral labels necessary for computing this error type.
• Figure 3: Distribution of bias attribution factor across model-mood tag combinations shows that gender bias can substantially affect downstream applications. The bias attribution factor $\hat{k}$ represents the percentage change in detected sentiment that can be attributed solely to gender composition differences between groups, assuming equal underlying emotional expression. The analysis shows that 48% of model $\times$ mood tag combinations have bias attribution factors exceeding 10%, which could meaningfully confound research conclusions.
• Figure S1: Distribution of posts across mood tags. The figure displays the proportion of posts labeled with a certain mood tag, where proportions are calculated within gender groups. Mood tags are ordered by their overall prevalence across all posts in the dataset.