Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffListener: Discrete Diffusion Model for Listener Generation

Siyeol Jung, Taehwan Kim

TL;DR

DiffListener addresses the listener head generation problem by removing autoregressive dependence and employing a discrete diffusion process over a fixed codebook learned by a VQ-VAE. It conditions listener motion on rich speaker cues, including facial expressions, audio, and text, enhanced with a novel facial differential modality to capture temporal rhythms. The approach combines a two-stage VQ-VAE quantization of listener motions with a VQ-Diffusion framework and a fusion network that integrates multimodal speaker information, enabling longer, identity-specific listener sequences with maintained coherence. Empirical results on Trevor and Stephen demonstrate state-of-the-art L2, FD, and P-FD metrics and favorable human judgments, while ablations confirm the value of differential facial information and textual context for natural, context-aware reactions.

Abstract

The listener head generation (LHG) task aims to generate natural nonverbal listener responses based on the speaker's multimodal cues. While prior work either rely on limited modalities (e.g. audio and facial information) or employ autoregressive approaches which have limitations such as accumulating prediction errors. To address these limitations, we propose DiffListener, a discrete diffusion based approach for non-autoregressive listener head generation. Our model takes the speaker's facial information, audio, and text as inputs, additionally incorporating facial differential information to represent the temporal dynamics of expressions and movements. With this explicit modeling of facial dynamics, DiffListener can generate coherent reaction sequences in a non-autoregressive manner. Through comprehensive experiments, DiffListener demonstrates state-of-the-art performance in both quantitative and qualitative evaluations. The user study shows that DiffListener generates natural context-aware listener reactions that are well synchronized with the speaker. The code and demo videos are available in https://siyeoljung.github.io/DiffListener

DiffListener: Discrete Diffusion Model for Listener Generation

TL;DR

DiffListener addresses the listener head generation problem by removing autoregressive dependence and employing a discrete diffusion process over a fixed codebook learned by a VQ-VAE. It conditions listener motion on rich speaker cues, including facial expressions, audio, and text, enhanced with a novel facial differential modality to capture temporal rhythms. The approach combines a two-stage VQ-VAE quantization of listener motions with a VQ-Diffusion framework and a fusion network that integrates multimodal speaker information, enabling longer, identity-specific listener sequences with maintained coherence. Empirical results on Trevor and Stephen demonstrate state-of-the-art L2, FD, and P-FD metrics and favorable human judgments, while ablations confirm the value of differential facial information and textual context for natural, context-aware reactions.

Abstract

The listener head generation (LHG) task aims to generate natural nonverbal listener responses based on the speaker's multimodal cues. While prior work either rely on limited modalities (e.g. audio and facial information) or employ autoregressive approaches which have limitations such as accumulating prediction errors. To address these limitations, we propose DiffListener, a discrete diffusion based approach for non-autoregressive listener head generation. Our model takes the speaker's facial information, audio, and text as inputs, additionally incorporating facial differential information to represent the temporal dynamics of expressions and movements. With this explicit modeling of facial dynamics, DiffListener can generate coherent reaction sequences in a non-autoregressive manner. Through comprehensive experiments, DiffListener demonstrates state-of-the-art performance in both quantitative and qualitative evaluations. The user study shows that DiffListener generates natural context-aware listener reactions that are well synchronized with the speaker. The code and demo videos are available in https://siyeoljung.github.io/DiffListener

Paper Structure

This paper contains 15 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Listener Head Generation
  4. Diffusion Model
  5. Methodology
  6. Problem Definition
  7. Quantizing Listener Motion
  8. Discrete Diffsuion Model
  9. Fusion Network
  10. Experiment
  11. Dataset
  12. Experimental Setup
  13. Quantitative Comparison
  14. Qualitative Comparision
  15. Conclusion

Figures (2)

  • Figure 1: The overview of DiffListener. The discrete diffusion model takes the fused representation and generates the listener's response sequence tokens. These tokens are then passed through the VQ-VAE decoder to obtain the listener's 3DMM coefficients.
  • Figure 2: The visualization comparison with baselines and ablated models at Trevor dataset. The blue and pink boxes include the speaker's utterance.