Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topic Bias in Emotion Classification

Maximilian Wegge, Roman Klinger

TL;DR

This work investigates topic bias as a confound in emotion classification, showing that sampling strategies inflect topic distributions that correlate with emotion labels across diverse corpora. It introduces a dual evaluation framework (InTopic vs CrossTopic) and uses BERTopic to automatically label topics, revealing corpus-specific topic–emotion associations. Debiasing via gradient reversal generally improves cross-corpus robustness, whereas simple word-removal can hurt performance in several settings, highlighting the nuanced trade-offs in mitigating topic signals. The findings underscore the need for more representative, topic-agnostic resources to fairly evaluate affective text classifiers and motivate future work on robust debiasing and few-shot approaches that resist topic-driven leakage.

Abstract

Emotion corpora are typically sampled based on keyword/hashtag search or by asking study participants to generate textual instances. In any case, these corpora are not uniform samples representing the entirety of a domain. We hypothesize that this practice of data acquisition leads to unrealistic correlations between overrepresented topics in these corpora that harm the generalizability of models. Such topic bias could lead to wrong predictions for instances like "I organized the service for my aunt's funeral." when funeral events are over-represented for instances labeled with sadness, despite the emotion of pride being more appropriate here. In this paper, we study this topic bias both from the data and the modeling perspective. We first label a set of emotion corpora automatically via topic modeling and show that emotions in fact correlate with specific topics. Further, we see that emotion classifiers are confounded by such topics. Finally, we show that the established debiasing method of adversarial correction via gradient reversal mitigates the issue. Our work points out issues with existing emotion corpora and that more representative resources are required for fair evaluation of models predicting affective concepts from text.

Topic Bias in Emotion Classification

TL;DR

This work investigates topic bias as a confound in emotion classification, showing that sampling strategies inflect topic distributions that correlate with emotion labels across diverse corpora. It introduces a dual evaluation framework (InTopic vs CrossTopic) and uses BERTopic to automatically label topics, revealing corpus-specific topic–emotion associations. Debiasing via gradient reversal generally improves cross-corpus robustness, whereas simple word-removal can hurt performance in several settings, highlighting the nuanced trade-offs in mitigating topic signals. The findings underscore the need for more representative, topic-agnostic resources to fairly evaluate affective text classifiers and motivate future work on robust debiasing and few-shot approaches that resist topic-driven leakage.

Abstract

Emotion corpora are typically sampled based on keyword/hashtag search or by asking study participants to generate textual instances. In any case, these corpora are not uniform samples representing the entirety of a domain. We hypothesize that this practice of data acquisition leads to unrealistic correlations between overrepresented topics in these corpora that harm the generalizability of models. Such topic bias could lead to wrong predictions for instances like "I organized the service for my aunt's funeral." when funeral events are over-represented for instances labeled with sadness, despite the emotion of pride being more appropriate here. In this paper, we study this topic bias both from the data and the modeling perspective. We first label a set of emotion corpora automatically via topic modeling and show that emotions in fact correlate with specific topics. Further, we see that emotion classifiers are confounded by such topics. Finally, we show that the established debiasing method of adversarial correction via gradient reversal mitigates the issue. Our work points out issues with existing emotion corpora and that more representative resources are required for fair evaluation of models predicting affective concepts from text.
Paper Structure (36 sections, 8 figures, 7 tables)

This paper contains 36 sections, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Emotion Classification
  4. Bias
  5. Detection and Mitigation.
  6. Methods & Experimental Setting
  7. Definitions.
  8. Topic-based Bias Detection
  9. Topic Modeling.
  10. Bias Mitigation
  11. Word Removal.
  12. Gradient Reversal.
  13. Data
  14. Results
  15. Are emotions biased towards topics?
  16. ...and 21 more sections

Figures (8)

  • Figure 1: Visualization of the experimental setting for InTopic and CrossTopic predictions.
  • Figure 2: Normalized pointwise mutual information between topics and emotion annotations in ISEAR.
  • Figure 3: Normalized pointwise mutual information between topics and emotion annotations in enVENT.
  • Figure 4: Normalized pointwise mutual information between topics and appraisal annotations in enVENT.
  • Figure 5: Micro-average F$_1$ for each topic-specific test set in ISEAR, for each held-out topic (CrossTopic/InTopic). No mitigation method is used (Bl setting).
  • ...and 3 more figures