VividListener: Expressive and Controllable Listener Dynamics Modeling for Multi-Modal Responsive Interaction

TL;DR

VividListener tackles the challenge of generating expressive, controllable listener head dynamics in long multi-turn dialogues by introducing ListenerX, a large-scale dataset with fine-grained multi-modal annotations, and a diffusion-based framework with a Responsive Interaction Module and Emotional Intensity Tags. The method integrates speaker cues and text-based guidance to produce coherent, emotionally rich listener reactions, with controllable intensity via EIT and AdaIN-based modulation. Extensive experiments show state-of-the-art performance on ListenerX, including strong realism, synchrony, and diversity, along with ablations validating the contribution of each component. The work advances practical dialogue modeling for virtual avatars and HCI by enabling long-range, multi-modal, and open-vocabulary control of listener behavior.

Abstract

Generating responsive listener head dynamics with nuanced emotions and expressive reactions is crucial for practical dialogue modeling in various virtual avatar animations. Previous studies mainly focus on the direct short-term production of listener behavior. They overlook the fine-grained control over motion variations and emotional intensity, especially in long-sequence modeling. Moreover, the lack of long-term and large-scale paired speaker-listener corpora including head dynamics and fine-grained multi-modality annotations (e.g., text-based expression descriptions, emotional intensity) also limits the application of dialogue modeling.Therefore, we first newly collect a large-scale multi-turn dataset of 3D dyadic conversation containing more than 1.4M valid frames for multi-modal responsive interaction, dubbed ListenerX. Additionally, we propose VividListener, a novel framework enabling fine-grained, expressive and controllable listener dynamics modeling. This framework leverages multi-modal conditions as guiding principles for fostering coherent interactions between speakers and listeners.Specifically, we design the Responsive Interaction Module (RIM) to adaptively represent the multi-modal interactive embeddings. RIM ensures the listener dynamics achieve fine-grained semantic coordination with textual descriptions and adjustments, while preserving expressive reaction with speaker behavior. Meanwhile, we design the Emotional Intensity Tags (EIT) for emotion intensity editing with multi-modal information integration, applying to both text descriptions and listener motion amplitude.Extensive experiments conducted on our newly collected ListenerX dataset demonstrate that VividListener achieves state-of-the-art performance, realizing expressive and controllable listener dynamics.

Figures (7)

• Figure 1: VividListener: listener dynamics modeling framework for multi-modal responsive interaction. This framework inputs speaker head motions, speaker audios, the conditions of listener textual descriptions, and the emotional intensity tags, which outputs listener head dynamics sequences. The generated listener head could change with the expression descriptions (from light-blue/disgusting to red/excited), and exhibit varying degrees of facial emotions (from light to dark).
• Figure 2: VividListener: listener dynamics modeling framework for multi-modal responsive interaction. This framework first integrates multi-modal inputs of speaker and listener through Responsive Interaction Module. Then, the fused features and conditions are fed into the DiT pipeline for the final listener dynamics modeling, which will change with the text descriptions and intensity tags, from cool(blue) tones to warm(red) tones. (Zoom in for better details.)
• Figure 3: Visual comparisons on ListenerX. We present visualizations of listener motions generated on ListenerX compared to various state-of-the-art methods. Unlike other approaches, our VividListener input incorporates fine-grained textual descriptions (shown in the last row).
• Figure 4: Conditional Control Results. We generate listener head dynamics by inputting the speaker audio and head motion information (shown in the top-left corner) alongside dataset-based emotion descriptions (shown in the bottom-left corner). Additionally, we utilize open-vocabulary text inputs to further enrich the generation process. Each example incorporates three VA values (e.g., -0.9, +0.3, +0.9) to control emotional intensity levels (Please Zoom in for better details.)
• Figure 5: User Study. Comparisons with SOTA methods.