Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spoken Conversational Agents with Large Language Models

Chao-Han Huck Yang, Andreas Stolcke, Larry Heck

TL;DR

This paper delivers a comprehensive tutorial-style synthesis of spoken conversational agents powered by large language models, tracing the evolution from cascaded ASR/NLU pipelines to end-to-end, cross-modal systems. It articulates theoretical and practical frameworks for adapting text LLMs to audio, aligning multimodal representations, and training joint speech-text models, while benchmarking datasets, metrics, and robustness across accents. The work contrasts cascaded versus end-to-end designs, discusses post-ASR correction and streaming considerations, and connects industrial assistants to both open-domain and task-oriented agents, outlining reproducible baselines and open problems in privacy, safety, and evaluation. It also provides actionable recipes, a structured tutorial outline, and a roadmap for future research in situated, multimodal dialogue systems, grounded in diverse user populations.

Abstract

Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We frame adaptation of text LLMs to audio, cross-modal alignment, and joint speech-text training; review datasets, metrics, and robustness across accents and compare design choices (cascaded vs. E2E, post-ASR correction, streaming). We link industrial assistants to current open-domain and task-oriented agents, highlight reproducible baselines, and outline open problems in privacy, safety, and evaluation. Attendees leave with practical recipes and a clear systems-level roadmap.

Spoken Conversational Agents with Large Language Models

TL;DR

This paper delivers a comprehensive tutorial-style synthesis of spoken conversational agents powered by large language models, tracing the evolution from cascaded ASR/NLU pipelines to end-to-end, cross-modal systems. It articulates theoretical and practical frameworks for adapting text LLMs to audio, aligning multimodal representations, and training joint speech-text models, while benchmarking datasets, metrics, and robustness across accents. The work contrasts cascaded versus end-to-end designs, discusses post-ASR correction and streaming considerations, and connects industrial assistants to both open-domain and task-oriented agents, outlining reproducible baselines and open problems in privacy, safety, and evaluation. It also provides actionable recipes, a structured tutorial outline, and a roadmap for future research in situated, multimodal dialogue systems, grounded in diverse user populations.

Abstract

Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We frame adaptation of text LLMs to audio, cross-modal alignment, and joint speech-text training; review datasets, metrics, and robustness across accents and compare design choices (cascaded vs. E2E, post-ASR correction, streaming). We link industrial assistants to current open-domain and task-oriented agents, highlight reproducible baselines, and outline open problems in privacy, safety, and evaluation. Attendees leave with practical recipes and a clear systems-level roadmap.

Paper Structure

This paper contains 12 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Tutorial Outline
  3. Language Modeling for Speech Processing Background, History, and Beyond
  4. Large Language Models for Audio, Speech, and Conversational Signals
  5. Spoken Dialogue Systems
  6. Tutorial Presenters
  7. Huck Yang
  8. Larry Heck
  9. Reading List and Prerequisite
  10. Breadth
  11. Diversity Considerations
  12. Ethics Statement

Figures (1)

  • Figure 1: Examples of spoken conversational agent with different LLMs to understand different linguistic information from (a) semantics, (b) paralinguistics, and to (c) more phonetic signals.