On the Use of Audio to Improve Dialogue Policies

TL;DR

This paper proposes new architectures to add audio information by combining speech and text embeddings using a Double Multi-Head Attention component, and shows that audio embedding-aware dialogue policies outperform text-based ones, particularly in noisy transcription scenarios.

Abstract

With the significant progress of speech technologies, spoken goal-oriented dialogue systems are becoming increasingly popular. One of the main modules of a dialogue system is typically the dialogue policy, which is responsible for determining system actions. This component usually relies only on audio transcriptions, being strongly dependent on their quality and ignoring very important extralinguistic information embedded in the user's speech. In this paper, we propose new architectures to add audio information by combining speech and text embeddings using a Double Multi-Head Attention component. Our experiments show that audio embedding-aware dialogue policies outperform text-based ones, particularly in noisy transcription scenarios, and that how text and audio embeddings are combined is crucial to improve performance. We obtained a 9.8% relative improvement in the User Request Score compared to an only-text-based dialogue system on the DSTC2 dataset.

Figures (2)

• Figure 1: Audio-aware dialogue policy architecture developed in AEA-2022.
• Figure 2: Enhanced audio-aware dialogue policy with Double Multi-Head Attention Multimodal Fusion.