Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction

Téo Guichoux, Théodor Lemerle, Shivam Mehta, Jonas Beskow, Gustav Eje Henter, Laure Soulier, Catherine Pelachaud, Nicolas Obin

TL;DR

Gelina tackles the lack of synchronized, naturalistic multimodal generation by jointly producing speech and co-speech gestures from text. It introduces an interleaved token autoregressive backbone with a separate gesture flow-matching decoder, enabling multi-speaker and gesture-style cloning and even gesture-only synthesis from speech. The method leverages large unimodal text–speech data for robust alignment and then fine-tunes on paired text–speech–gesture data to achieve synchronized outputs. Experiments on BEAT2 show competitive speech quality and superior gesture realism compared with unimodal baselines, with near real-time performance.

Abstract

Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.

Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction

TL;DR

Gelina tackles the lack of synchronized, naturalistic multimodal generation by jointly producing speech and co-speech gestures from text. It introduces an interleaved token autoregressive backbone with a separate gesture flow-matching decoder, enabling multi-speaker and gesture-style cloning and even gesture-only synthesis from speech. The method leverages large unimodal text–speech data for robust alignment and then fine-tunes on paired text–speech–gesture data to achieve synchronized outputs. Experiments on BEAT2 show competitive speech quality and superior gesture realism compared with unimodal baselines, with near real-time performance.

Abstract

Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
Paper Structure (11 sections, 4 equations, 5 figures, 1 table)

This paper contains 11 sections, 4 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. GELINA architecture
  4. Text, Speech, and Gesture Tokenization
  5. Autoregressive Backbone
  6. Conditional flow-matching
  7. Experiments
  8. Experimental setting
  9. Evaluation and results
  10. Conclusion and future work
  11. Compliance with Ethical Standards

Figures (5)

  • Figure 2: Mean Opinion Scores with 95% confidence intervals from user evaluation across three aspects: Voice human-likeness, speech-gesture synchrony, and Gesture human-likeness.
  • Figure : (a) Gesture tokenizer.
  • Figure : (a) Gesture tokenizer.
  • Figure : (b) Speech and gesture autoregressive backbone.
  • Figure : (c) Gesture conditional flow-matching decoder.