What You Feel Is Not What They See: On Predicting Self-Reported Emotion from Third-Party Observer Labels
Yara El-Tawil, Aneesha Sampath, Emily Mower Provost
TL;DR
This paper investigates whether models trained on third-party emotion labels can predict self-reported emotions across corpora. Using MSP-Podcast, MuSE, and IEMOCAP with audio and language transformers and large language models, the authors quantify activation and valence predictability via the Concordance Correlation Coefficient. They find activation is largely unpredictable ($CCC \\approx 0$), valence is moderately predictable ($CCC \\approx 0.3$), and crucially, when content carries personal significance, third-party models achieve strong valence alignment ($CCC \\approx 0.6-0.8$). The results suggest that personal significance is a key pathway to aligning external perception with internal experience, with important implications for self-report modeling and dataset development.
Abstract
Self-reported emotion labels capture internal experience, while third-party labels reflect external perception. These perspectives often diverge, limiting the applicability of third-party-trained models to self-report contexts. This gap is critical in mental health, where accurate self-report modeling is essential for guiding intervention. We present the first cross-corpus evaluation of third-party-trained models on self-reports. We find activation unpredictable (CCC approximately 0) and valence moderately predictable (CCC approximately 0.3). Crucially, when content is personally significant to the speaker, models achieve high performance for valence (CCC approximately 0.6-0.8). Our findings point to personal significance as a key pathway for aligning external perception with internal experience and underscore the challenge of self-report activation modeling.