Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models

Pavel Stepachev, Pinzhen Chen, Barry Haddow

TL;DR

It is shown that the conversation context has diminishing returns and the metric used to select the transcript for prediction is crucial, and the best submission surpasses the provided baseline by 20% in absolute accuracy.

Abstract

Large language models (LLMs) have started to play a vital role in modelling speech and text. To explore the best use of context and multiple systems' outputs for post-ASR speech emotion prediction, we study LLM prompting on a recent task named GenSEC. Our techniques include ASR transcript ranking, variable conversation context, and system output fusion. We show that the conversation context has diminishing returns and the metric used to select the transcript for prediction is crucial. Finally, our best submission surpasses the provided baseline by 20% in absolute accuracy.

Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models

TL;DR

It is shown that the conversation context has diminishing returns and the metric used to select the transcript for prediction is crucial, and the best submission surpasses the provided baseline by 20% in absolute accuracy.

Abstract

Large language models (LLMs) have started to play a vital role in modelling speech and text. To explore the best use of context and multiple systems' outputs for post-ASR speech emotion prediction, we study LLM prompting on a recent task named GenSEC. Our techniques include ASR transcript ranking, variable conversation context, and system output fusion. We show that the conversation context has diminishing returns and the metric used to select the transcript for prediction is crucial. Finally, our best submission surpasses the provided baseline by 20% in absolute accuracy.
Paper Structure (18 sections, 1 equation, 4 figures, 5 tables)

This paper contains 18 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. DATA AND EVALUATION
  3. CONTEXT SELECTION METHODOLOGY
  4. Ranking
  5. Naive selection heuristics
  6. The use of conversation context
  7. ASR output fusion
  8. EXPERIMENTS AND RESULTS
  9. Setup
  10. Results
  11. Comparison between GPT models
  12. Effects of the conversation context
  13. ASR output ranking
  14. ASR-context fusion
  15. Final results
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Prompt template with context size 2 with the last utterance needing emotion prediction.
  • Figure 2: Prompt template with a context size 4 as well as 5 ASR outputs as a means of fusion.
  • Figure 3: Performance of ranking metrics with various context sizes on gpt-4o.
  • Figure 4: Performance of naive heuristics metrics with various context sizes on gpt-4o.