MASIVE: Open-Ended Affective State Identification in English and Spanish

TL;DR

This work collects and publishes MASIVE, a dataset of Reddit posts in English and Spanish containing over 1,000 unique affective states each, and defines the new problem of affective state identification for language generation models framed as a masked span prediction task.

Abstract

In the field of emotion analysis, much NLP research focuses on identifying a limited number of discrete emotion categories, often applied across languages. These basic sets, however, are rarely designed with textual data in mind, and culture, language, and dialect can influence how particular emotions are interpreted. In this work, we broaden our scope to a practically unbounded set of \textit{affective states}, which includes any terms that humans use to describe their experiences of feeling. We collect and publish MASIVE, a dataset of Reddit posts in English and Spanish containing over 1,000 unique affective states each. We then define the new problem of \textit{affective state identification} for language generation models framed as a masked span prediction task. On this task, we find that smaller finetuned multilingual models outperform much larger LLMs, even on region-specific Spanish affective states. Additionally, we show that pretraining on MASIVE improves model performance on existing emotion benchmarks. Finally, through machine translation experiments, we find that native speaker-written data is vital to good performance on this task.

Figures (5)

• Figure 1: Paraphrased input and expected output examples from MASIVE in English and Spanish. Models are tasked with predicting affective states (highlighted), which reflect more nuanced feelings than label sets in prior work, such as the Ekman basic emotions.
• Figure 2: Illustration of the bootstrapping procedure used to collect texts and automatically extracted affective state labels in the MASIVE corpus.
• Figure 3: Top-k accuracy and similarity results on subsets reflecting different linguistic constructions in MASIVE: grammatical gender of affective states in Spanish (left) and negated expressions in Spanish (center) and English (right). Shades reflect different values of k separated by small gaps, where the lightest shade represents $k=1$ and the darkest shade represents $k=5$.
• Figure 4: Instructions provided to our human annotators, including definitions. Annotators may collapse or expand the instructions at will.
• Figure 5: Human annotation interface with a sample datapoint. Clicking the button to show more or less context toggles the display of the full Reddit post vs. the one-sentence context. As shown, the Emotion/Mood and Figurative Language questions only appear if the highlighted term is judged like an affective state or completely an affective state.