Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Translating Emotions to Annotations -- A Participant Perspective of Physiological Emotion Data Collection

Pragya Singh, Ritvik Budhiraja, Pankaj Jalote, Mohan Kumar, Pushpendra Singh

TL;DR

This work addresses the challenge of aligning physiological signals with emotional states by examining how participant subjectivity and data-collection design affect annotations. It conducts a lab VR study with 37 participants, recording PPG and EDA while collecting self-reports, followed by semi-structured interviews to uncover participant perspectives. The findings reveal that participant perception, stimulus content, and experiment design heavily shape annotations, often beyond what physiology alone indicates, highlighting a gap between subjective reports and objective signals. The authors propose participant-centric design guidelines, richer contextual labeling, and pathways toward annotation-free approaches to improve data quality for AI emotion recognition, with practical implications for CSCW, HCI, and AI communities.

Abstract

Physiological signals hold immense potential for ubiquitous emotion monitoring, presenting numerous applications in emotion recognition. However, harnessing this potential is hindered by significant challenges, particularly in the collection of annotations that align with physiological changes since the process hinges heavily on human participants. In this work, we set out to study human participant perspectives in the emotion data collection procedure. We conducted a lab-based emotion data collection study with 37 participants using 360 degree virtual reality video stimulus followed by semi-structured interviews with the study participants. Our findings presented that intrinsic factors like participant perception, experiment design nuances, and experiment setup suitability impact their emotional response and annotation within lab settings. Drawing from our findings and prior research, we propose recommendations for incorporating participant context into annotations and emphasizing participant-centric experiment designs. Furthermore, we explore current emotion data collection practices followed by AI practitioners and offer insights for future contributions leveraging physiological emotion data.

Translating Emotions to Annotations -- A Participant Perspective of Physiological Emotion Data Collection

TL;DR

This work addresses the challenge of aligning physiological signals with emotional states by examining how participant subjectivity and data-collection design affect annotations. It conducts a lab VR study with 37 participants, recording PPG and EDA while collecting self-reports, followed by semi-structured interviews to uncover participant perspectives. The findings reveal that participant perception, stimulus content, and experiment design heavily shape annotations, often beyond what physiology alone indicates, highlighting a gap between subjective reports and objective signals. The authors propose participant-centric design guidelines, richer contextual labeling, and pathways toward annotation-free approaches to improve data quality for AI emotion recognition, with practical implications for CSCW, HCI, and AI communities.

Abstract

Physiological signals hold immense potential for ubiquitous emotion monitoring, presenting numerous applications in emotion recognition. However, harnessing this potential is hindered by significant challenges, particularly in the collection of annotations that align with physiological changes since the process hinges heavily on human participants. In this work, we set out to study human participant perspectives in the emotion data collection procedure. We conducted a lab-based emotion data collection study with 37 participants using 360 degree virtual reality video stimulus followed by semi-structured interviews with the study participants. Our findings presented that intrinsic factors like participant perception, experiment design nuances, and experiment setup suitability impact their emotional response and annotation within lab settings. Drawing from our findings and prior research, we propose recommendations for incorporating participant context into annotations and emphasizing participant-centric experiment designs. Furthermore, we explore current emotion data collection practices followed by AI practitioners and offer insights for future contributions leveraging physiological emotion data.

Paper Structure

This paper contains 33 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Data-Work and Emotion Data
  4. Emotion Data Collection Practices
  5. Methodology
  6. Participant Selection
  7. Stimulus Selection and Data Collection
  8. Experiment Setup
  9. Interviews
  10. Data Analysis
  11. Study Limitations
  12. Findings
  13. Unveiling the Influence of Participant Perception
  14. The Influence of Participant's Self-perception
  15. The Influence of Perception towards the Content of Stimulus
  16. ...and 18 more sections

Figures (3)

  • Figure 1: The figure depicts the data collection procedure used in this paper in their execution order. In the diagram, the following abbreviations correspond to the respective components: GHQ (General Health Questionnaire), BFI10 (Big Five Inventory-10), VRSQ (Virtual Reality Sickness Questionnaire), PPG (Photoplethysmography), EDA (Galvanic Skin Response), V (video or Stimulus), and R (Rest). The Pre and Post-Exposure Ratings include the Positive-Negative Affect Scale (PANAS) and the SAM Scales.
  • Figure 2: The figure shows stills taken from the $360^{\circ}$ videos capturing different environments shown to the participants as a part of the experiment methodology. The stills capture different valence-arousal combinations such as (a) Low-Valence-High-Arousal (LVHA), (b) High-Valence-High-Arousal (HVHA), (c) High-Valence-low-arousal (HVLA) and (d) Low-Valence-Low-Arousal (LVLA), the database is publically available li2017public.
  • Figure 3: Study Design