Towards Data Distillation for End-to-end Spoken Conversational Question Answering

TL;DR

This work introduces Spoken-CoQA, a large-scale dataset for spoken conversational question answering, and DDNet, a data distillation framework that fuses audio and text features to mitigate ASR errors in end-to-end QA. By employing Speech-BERT and Text-BERT with cross-attention and a teacher-student knowledge distillation mechanism, the approach improves robustness when handling ASR transcripts. Experiments across multiple baselines demonstrate consistent gains in F1/EM on Spoken-CoQA, highlighting the value of joint audio-text modeling and distillation for multi-turn spoken QA. The work advances practical spoken dialogue systems by bridging speech and text representations in a unified QA framework.

Abstract

In spoken question answering, QA systems are designed to answer questions from contiguous text spans within the related speech transcripts. However, the most natural way that human seek or test their knowledge is via human conversations. Therefore, we propose a new Spoken Conversational Question Answering task (SCQA), aiming at enabling QA systems to model complex dialogues flow given the speech utterances and text corpora. In this task, our main objective is to build a QA system to deal with conversational questions both in spoken and text forms, and to explore the plausibility of providing more cues in spoken documents with systems in information gathering. To this end, instead of adopting automatically generated speech transcripts with highly noisy data, we propose a novel unified data distillation approach, DDNet, which directly fuse audio-text features to reduce the misalignment between automatic speech recognition hypotheses and the reference transcriptions. In addition, to evaluate the capacity of QA systems in a dialogue-style interaction, we assemble a Spoken Conversational Question Answering (Spoken-CoQA) dataset with more than 120k question-answer pairs. Experiments demonstrate that our proposed method achieves superior performance in spoken conversational question answering.

Figures (4)

• Figure 1: An illustration of flow diagram for spoken conversational question answering tasks with an example from our proposed Spoken-CoQA dataset.
• Figure 2: An illustration of the architecture of DDNet.
• Figure 3: Ablation studies of temperature $\tau$ on DDNet performance (FlowQA, SDNet, BERT, ALBERT). Red and blue denote the results on CoQA dev and Spoken-CoQA test set, respectively.
• Figure 4: Examples of the log-mel spectrograms and the corresponding MFCC feature embedding. It can see that the log-mel spectrograms corresponds to different example sentences from the Spoken-CoQA dataset.