Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems

Hitesh Tulsiani, David M. Chan, Shalini Ghosh, Garima Lalwani, Prabhat Pandey, Ankish Bansal, Sri Garimella, Ariya Rastrow, Björn Hoffmeister

TL;DR

This work introduces a general framework for ASR in dialog systems that can go beyond learning from single-turn utterances and learn over time how to adapt to both explicit supervision and implicit user feedback present in multi-turn conversations.

Abstract

Dialog systems, such as voice assistants, are expected to engage with users in complex, evolving conversations. Unfortunately, traditional automatic speech recognition (ASR) systems deployed in such applications are usually trained to recognize each turn independently and lack the ability to adapt to the conversational context or incorporate user feedback. In this work, we introduce a general framework for ASR in dialog systems that can go beyond learning from single-turn utterances and learn over time how to adapt to both explicit supervision and implicit user feedback present in multi-turn conversations. We accomplish that by leveraging advances in student-teacher learning and context-aware dialog processing, and designing contrastive self-supervision approaches with Ohm, a new online hard-negative mining approach. We show that leveraging our new framework compared to traditional training leads to relative WER reductions of close to 10% in real-world dialog systems, and up to 26% on public synthetic data.

An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems

TL;DR

This work introduces a general framework for ASR in dialog systems that can go beyond learning from single-turn utterances and learn over time how to adapt to both explicit supervision and implicit user feedback present in multi-turn conversations.

Abstract

Dialog systems, such as voice assistants, are expected to engage with users in complex, evolving conversations. Unfortunately, traditional automatic speech recognition (ASR) systems deployed in such applications are usually trained to recognize each turn independently and lack the ability to adapt to the conversational context or incorporate user feedback. In this work, we introduce a general framework for ASR in dialog systems that can go beyond learning from single-turn utterances and learn over time how to adapt to both explicit supervision and implicit user feedback present in multi-turn conversations. We accomplish that by leveraging advances in student-teacher learning and context-aware dialog processing, and designing contrastive self-supervision approaches with Ohm, a new online hard-negative mining approach. We show that leveraging our new framework compared to traditional training leads to relative WER reductions of close to 10% in real-world dialog systems, and up to 26% on public synthetic data.
Paper Structure (32 sections, 2 figures, 9 tables, 1 algorithm)

This paper contains 32 sections, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Supervised Context Modeling
  4. Unsupervised Learning From Dialogue Contexts
  5. Model Distillation
  6. Self-Learning for Dialogue ASR
  7. Teacher Model
  8. Learning from Explicit Context
  9. Audio context modeling
  10. Text context modeling
  11. Learning from Implicit Context
  12. Online Hard Negative Mining (Ohm)
  13. Reformulation Up-Sampling
  14. Student-Teacher Distillation
  15. Experimental Design
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Traditional dialog systems learn to perform ASR using only supervised feedback with large-scale unsupervised/semi-supervised pre-training on single isolated utterances. This work introduces a novel general framework leveraging student-teacher distillation, contrastive learning, and online hard-negative mining, allowing ASR systems to learn from contextual clues and implicit feedback present in full conversational transcripts. Our two stage system naturally allows us to distill contextual signals from a context-aware teacher model to a context unaware student model.
  • Figure 2: An overview of our approach. During training our teacher model ingests context from past/future audio and text along with the current utterance, and learns both implicitly and explicitly using CLC chan2024task for implicit context learning and supervised joint loss for explicit learning from supervised data. $\uparrow$ show data flow in forward-pass, and $\downarrow$$\downarrow$ show loss propagation from each of the components.