AV-Dialog: Spoken Dialogue Models with Audio-Visual Input

TL;DR

AV-Dialog tackles the cocktail-party problem by introducing a streaming audio-visual dialogue framework that fuses audio and visual cues to track the target speaker, predict turn-taking, and produce coherent responses. It leverages DAC acoustic tokens and AV-HuBERT visual features within a two-branch (dual-model) or single-branch (unified) architecture, trained through a two-stage, multi-task regimen plus synthetic mixing to improve robustness in noisy, multi-speaker settings. Key contributions include explicit turn-taking supervision, a dedicated AVSR-aligned output stream, and comprehensive evaluations showing superior transcription accuracy, turn-taking timing, and dialogue quality compared with audio-only baselines. The work demonstrates that integrating seeing and hearing yields speaker-aware interactions with practical potential for robust, real-world spoken dialogue agents in noisy environments.

Abstract

Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.

Figures (9)

• Figure 1: AV-Dialog understands audio-visual input from the target user (purple waveform), accurately detects the appropriate time to take a turn in the conversation, and outputs responses (blue waveform), even in the presence of interfering speakers (brown waveform).
• Figure 2: Token sequence and dual-model design. A. We use a DAC tokenizer to encode audio into 16 audio token streams and use AV-HuBERT to convert video to continuous visual features. We use turn-level annotation and word-level alignment to generate the target output text stream. B. shows our dual-model pipeline for AV-dialog. The AV dialogue understanding module recognizes user speech and detects potential turn-taking events, while the text backbone generates high-quality responses once turn-taking is triggered.
• Figure 3: Unified AV-Dialog model. It takes the audio-visual input and predicts the turn-taking events. When the special turn-taking token is generated, the AV-Dialog model generates the response on the same output stream.
• Figure 4: Model performance across different SNRs. Plot A shows the WER of streaming AVSR task on different SNRs of noisy audio input. Plot B shows the response ratio of the turn-taking prediction on different SNRs of noisy audio input.
• Figure 5: Training input and output tokens design for different tasks at Stage 1.