Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Affect Mining Techniques for Annotation Sample Selection in the Creation of Finnish Affective Speech Corpus

Kalle Lahtinen, Einari Vaaras, Liisa Mustanoja, Okko Räsänen

TL;DR

This work addresses the lack of spontaneous Finnish affective speech data for affective speech recognition by constructing FinnAffect, the first such corpus for Finnish. It employs medoid-based active learning (MAL) across three large Finnish corpora (LP, TP, HP) and integrates acoustic (eGeMAPS), cross-linguistic SER (ExHuBERT), and text sentiment (FinnSentiment) features to select 12,000 samples for annotation, including 9,000 MAL-derived and 3,000 random samples. Although MAL did not outperform random sampling in overall diversity for the full dataset, post-hoc analyses reveal that FAFT and FAFT-initialized k-medoids, as well as MATLAB CLARA variants, can improve affective diversity for subsets up to around 1,500 samples, with feature choices (ExHuBERT and FinnSentiment) boosting informative dimensions like valence. The resulting FinnAffect corpus, along with the methodological insights, provides a baseline for future cross-language affective corpus creation and informs sampling strategies for SER development in Finnish and other languages.

Abstract

Study of affect in speech requires suitable data, as emotional expression and perception vary across languages. Until now, no corpus has existed for natural expression of affect in spontaneous Finnish, existing data being acted or from a very specific communicative setting. This paper presents the first such corpus, created by annotating 12,000 utterances for emotional arousal and valence, sampled from three large-scale Finnish speech corpora. To ensure diverse affective expression, sample selection was conducted with an affect mining approach combining acoustic, cross-linguistic speech emotion, and text sentiment features. We compare this method to random sampling in terms of annotation diversity, and conduct post-hoc analyses to identify sampling choices that would have maximized the diversity. As an outcome, the work introduces a spontaneous Finnish affective speech corpus and informs sampling strategies for affective speech corpus creation in other languages or domains.

Investigating Affect Mining Techniques for Annotation Sample Selection in the Creation of Finnish Affective Speech Corpus

TL;DR

This work addresses the lack of spontaneous Finnish affective speech data for affective speech recognition by constructing FinnAffect, the first such corpus for Finnish. It employs medoid-based active learning (MAL) across three large Finnish corpora (LP, TP, HP) and integrates acoustic (eGeMAPS), cross-linguistic SER (ExHuBERT), and text sentiment (FinnSentiment) features to select 12,000 samples for annotation, including 9,000 MAL-derived and 3,000 random samples. Although MAL did not outperform random sampling in overall diversity for the full dataset, post-hoc analyses reveal that FAFT and FAFT-initialized k-medoids, as well as MATLAB CLARA variants, can improve affective diversity for subsets up to around 1,500 samples, with feature choices (ExHuBERT and FinnSentiment) boosting informative dimensions like valence. The resulting FinnAffect corpus, along with the methodological insights, provides a baseline for future cross-language affective corpus creation and informs sampling strategies for SER development in Finnish and other languages.

Abstract

Study of affect in speech requires suitable data, as emotional expression and perception vary across languages. Until now, no corpus has existed for natural expression of affect in spontaneous Finnish, existing data being acted or from a very specific communicative setting. This paper presents the first such corpus, created by annotating 12,000 utterances for emotional arousal and valence, sampled from three large-scale Finnish speech corpora. To ensure diverse affective expression, sample selection was conducted with an affect mining approach combining acoustic, cross-linguistic speech emotion, and text sentiment features. We compare this method to random sampling in terms of annotation diversity, and conduct post-hoc analyses to identify sampling choices that would have maximized the diversity. As an outcome, the work introduces a spontaneous Finnish affective speech corpus and informs sampling strategies for affective speech corpus creation in other languages or domains.

Paper Structure

This paper contains 10 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Prior work
  3. Creation of the FinnAffect corpus
  4. Data sources
  5. Sample selection with affect mining
  6. Annotation process
  7. Corpus description
  8. Post-hoc analysis of affect mining
  9. Conclusions
  10. Acknowledgements

Figures (3)

  • Figure 1: Annotated arousal and valence scores on the GS subset (2,000 samples with 5 annotations), sorted by the mean across annotators. Shading denotes $\pm1$ SD across the annotators.
  • Figure 2: Standard deviation (SD) of annotator scores for valence (top) and arousal (bottom) in case of different simulated sample selection strategies for annotation, and as a function of number of samples chosen. Shading denotes $\pm1$ standard error across different runs/samplings.
  • Figure 3: The results of the cluster purity experiments for the four feature types, averaged across all 16 clustering algorithm variants (standard error as shaded area). The results are shown for valence (top) and arousal (bottom).