Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

TL;DR

USDM integrates speech and text within a single LLM framework to produce prosody-rich spoken dialog without separate ASR/TTS stages. A unified cross-modal pretraining scheme with a 10{,}000-token acoustic-unit vocabulary and two special tokens enables end-to-end generation of speech and text, aided by a unit-to-speech decoder based on Voicebox and a BigVGAN vocoder. Evaluations on DailyTalk with automatic metrics and human judgments show superior semantic coherence and natural prosody compared to cascaded and end-to-end baselines, validating the importance of cross-modal pretraining and multi-step dialog templates. The approach lays a foundation for scalable, voice-first conversational systems and motivates multilingual extensions and direct speech generation in future work.

Abstract

Recent work shows promising results in expanding the capabilities of large language models (LLM) to directly understand and synthesize speech. However, an LLM-based strategy for modeling spoken dialogs remains elusive, calling for further investigation. This paper introduces an extensive speech-text LLM framework, the Unified Spoken Dialog Model (USDM), designed to generate coherent spoken responses with naturally occurring prosodic features relevant to the given input speech without relying on explicit automatic speech recognition (ASR) or text-to-speech (TTS) systems. We have verified the inclusion of prosody in speech tokens that predominantly contain semantic information and have used this foundation to construct a prosody-infused speech-text model. Additionally, we propose a generalized speech-text pretraining scheme that enhances the capture of cross-modal semantics. To construct USDM, we fine-tune our speech-text model on spoken dialog data using a multi-step spoken dialog template that stimulates the chain-of-reasoning capabilities exhibited by the underlying LLM. Automatic and human evaluations on the DailyTalk dataset demonstrate that our approach effectively generates natural-sounding spoken responses, surpassing previous and cascaded baselines. Our code and checkpoints are available at https://github.com/naver-ai/usdm.

Figures (8)

• Figure 1: Overview of our spoken dialog modeling approach (Left). All possible self-supervised learning objectives from our speech-text pretraining scheme. (Right)
• Figure 2: Pitch contour of the original audio and the audio reconstructed from extracted acoustic units. Due to the stochastic nature of the reconstruction model, we attempt reconstruction twice, demonstrating that the pitch variation closely mirrors the ground truth.
• Figure 3: The overall speech-text pretraining scheme.
• Figure 4: Attention maps between the generated responses of the USDM and the input speech (s) and its transcribed text (t). Input speech: "Oh, I can't believe it. He looks very young."
• Figure 5: Fine-tuning template for single-turn spoken dialog modeling. Left is the template used for training spoken dialog models (USDM, From Scratch), while the right is the template for training a text dialog model (Cascaded).