OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

TL;DR

The study defines Online Multimodal Conversational Response Generation (OMCRG) and introduces OmniResponse, a multimodal LLM that jointly generates synchronized listener text, facial cues, and audio in real time. It tackles temporal alignment by using Chrono-Text Markup and a TempoVoice synthesis pathway, enabling end-to-end online generation conditioned on speaker multimodal inputs. To support research, the authors release ResponseNet, a 696-sample, richly annotated dyadic dataset with synchronized video, separated audio channels, transcripts, and facial behavior annotations. Empirical results show OmniResponse outperforms baselines on semantic content, synchronization, and perceptual quality, supported by ablations and a user study, and the work provides a clear path for future exploration in online multimodal dialogue systems.

Abstract

In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), a novel task designed to produce synchronized verbal and non-verbal listener feedback online, based on the speaker's multimodal inputs. OMCRG captures natural dyadic interactions and introduces new challenges in aligning generated audio with listeners' facial responses. To tackle these challenges, we incorporate text as an intermediate modality to connect audio and facial responses. We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses. OmniResponse leverages a pretrained LLM enhanced with two core components: Chrono-Text Markup, which precisely timestamps generated text tokens, and TempoVoice, a controllable online text-to-speech (TTS) module that outputs speech synchronized with facial responses. To advance OMCRG research, we offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors. Comprehensive evaluations on ResponseNet demonstrate that OmniResponse outperforms baseline models in terms of semantic speech content, audio-visual synchronization, and generation quality. Our dataset, code, and models are publicly available.

Figures (9)

• Figure 1: Illustration of the new OMCRG task. (a) In offline tasks, the generation model generates the listener’s full response only after receiving the entire input sequence from the speaker. (b) Differently, OMCRG task requires sequentially processing the speaker’s incoming input and generating multi-modal responses for the listener on the fly.
• Figure 2: Overview of the proposed OmniResponse. The model takes textual conversational history and newly coming multimodal information (e.g., facial cues) from the speaker and listener as input, and generates temporally synchronized facial and textual responses for the listener by leveraging a pre-trained LLM enhanced with our proposed Chrono-Text Markup. The generated text embeddings are converted into audio synchronized with the facial response by the proposed TempoVoice module.
• Figure 3: Architecture of TempoVoice. TempoVoice transforms textual hidden‐state embeddings into audio segments.
• Figure 4: Statistics of ResponseNet. (a) Distribution of video clip durations. (b) Distribution of dyadic conversation topics. (c) Word cloud of spoken words in dyadic conversations.
• Figure 5: Qualitative Results. Given the speaker’s audio and video streams and corresponding utterances (left), OmniResponse autoregressively generates synchronized visual, audio, and textual response streams (right). For clarity, [LASTING] tokens are removed from the generated dialogue.