Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback

TL;DR

This paper tackles the challenge of optimizing conversational quality in spoken dialogue systems by introducing the first multi-reward RLAIF framework that jointly targets semantic coherence, audio naturalness, intelligibility, and emotion-consistency. It operationalizes preference learning via Direct Preference Optimization (DPO) for both turn-by-turn and blockwise duplex SDS, and aligns utterance-level rewards with partial, streaming generation through blockwise log-probability aggregation. A large-scale, multi-reward preference dataset (165.7K pairs) supports training, with experiments showing that single-reward improvements are operation-specific while joint rewards yield consistent gains across semantic and acoustic metrics. The work demonstrates significant improvements in LLM-judge coherence, audio quality, and intelligibility, and provides public release of data and code to enable reproducible research in holistic SDS alignment. Overall, this approach moves SDS toward human-aligned, natural, and emotionally coherent interactions in real-time, duplex settings.

Abstract

Reinforcement learning from human or AI feedback (RLHF/RLAIF) for speech-in/speech-out dialogue systems (SDS) remains underexplored, with prior work largely limited to single semantic rewards applied at the utterance level. Such setups overlook the multi-dimensional and multi-modal nature of conversational quality, which encompasses semantic coherence, audio naturalness, speaker consistency, emotion alignment, and turn-taking behavior. Moreover, they are fundamentally mismatched with duplex spoken dialogue systems that generate responses incrementally, where agents must make decisions based on partial utterances. We address these limitations with the first multi-reward RLAIF framework for SDS, combining semantic, audio-quality, and emotion-consistency rewards. To align utterance-level preferences with incremental, blockwise decoding in duplex models, we apply turn-level preference sampling and aggregate per-block log-probabilities within a single DPO objective. We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models, and release a multi-reward DPO dataset to support reproducible research. Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness. These results highlight the importance of holistic, multi-reward alignment for practical conversational SDS.

Figures (3)

• Figure 1: Overview of the proposed dataset-level Multi-Reward RLAIF framework. Unlike prior work that optimizes single rewards (typically semantic quality), we construct four independent preference datasets targeting semantic coherence, audio naturalness, intelligibility, and emotion consistency. These datasets are jointly sampled during DPO training. The framework supports both turn-by-turn CoT and blockwise duplex SDS by aggregating per-block log-probabilities under utterance-level preferences (S. \ref{['subsec:blockwise_dpo']})
• Figure 2: Distribution of LLM judge score over candidate responses
• Figure 3: Distribution of AutoBLEU score over candidate responses