Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers

Kyeongman Park, Seongho Joo, Kyomin Jung

TL;DR

MultiActor-Audiobook tackles zero-shot audiobook generation for stories with multiple speakers by introducing Multimodal Speaker Persona Generation and LLM-Based Script Instruction Generation. By pairing LLM-derived character personas with photorealistic faces and Face-to-Voice synthesis, plus sentence-level emotional instructions guided by context, the system achieves speaker-consistent and emotionally expressive narration without additional training. evaluations against baselines show competitive human MOS and superior MLLM performance, with ablations confirming the value of both MSP and LS I. The approach reduces data and annotation costs while delivering natural, context-aware prosody, facilitating scalable, multimodal audiobook creation.

Abstract

We introduce MultiActor-Audiobook, a zero-shot approach for generating audiobooks that automatically produces consistent, expressive, and speaker-appropriate prosody, including intonation and emotion. Previous audiobook systems have several limitations: they require users to manually configure the speaker's prosody, read each sentence with a monotonic tone compared to voice actors, or rely on costly training. However, our MultiActor-Audiobook addresses these issues by introducing two novel processes: (1) MSP (**Multimodal Speaker Persona Generation**) and (2) LSI (**LLM-based Script Instruction Generation**). With these two processes, MultiActor-Audiobook can generate more emotionally expressive audiobooks with a consistent speaker prosody without additional training. We compare our system with commercial products, through human and MLLM evaluations, achieving competitive results. Furthermore, we demonstrate the effectiveness of MSP and LSI through ablation studies.

MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers

TL;DR

MultiActor-Audiobook tackles zero-shot audiobook generation for stories with multiple speakers by introducing Multimodal Speaker Persona Generation and LLM-Based Script Instruction Generation. By pairing LLM-derived character personas with photorealistic faces and Face-to-Voice synthesis, plus sentence-level emotional instructions guided by context, the system achieves speaker-consistent and emotionally expressive narration without additional training. evaluations against baselines show competitive human MOS and superior MLLM performance, with ablations confirming the value of both MSP and LS I. The approach reduces data and annotation costs while delivering natural, context-aware prosody, facilitating scalable, multimodal audiobook creation.

Abstract

We introduce MultiActor-Audiobook, a zero-shot approach for generating audiobooks that automatically produces consistent, expressive, and speaker-appropriate prosody, including intonation and emotion. Previous audiobook systems have several limitations: they require users to manually configure the speaker's prosody, read each sentence with a monotonic tone compared to voice actors, or rely on costly training. However, our MultiActor-Audiobook addresses these issues by introducing two novel processes: (1) MSP (**Multimodal Speaker Persona Generation**) and (2) LSI (**LLM-based Script Instruction Generation**). With these two processes, MultiActor-Audiobook can generate more emotionally expressive audiobooks with a consistent speaker prosody without additional training. We compare our system with commercial products, through human and MLLM evaluations, achieving competitive results. Furthermore, we demonstrate the effectiveness of MSP and LSI through ablation studies.
Paper Structure (21 sections, 1 figure, 2 tables)

This paper contains 21 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. MultiActor-Audiobook
  3. Multimodal Speaker Persona Generation
  4. Extract all speakers and their characteristics from story
  5. Create facial personas
  6. Create audio personas
  7. LLM-Based Script Instruction Generation
  8. Identify the speaker of each sentence
  9. Create Text Description of Each Sentence
  10. Integration All Input Data
  11. Experiments
  12. Experimental Setup
  13. Story Dataset
  14. MultiActor-Audiobook Implementation Details
  15. Baselines
  16. ...and 6 more sections

Figures (1)

  • Figure 1: The MultiActor-Audiobook. At the left side of figure (a), we perform Multimodal Speaker Persona Generation to create each speaker's AI-generated face images and voice samples, and LLM-based Script Instruction Generation to annotate every sentence's speaking instructions. At the right side of figure (b), we input all the multimodal input to our backbone TTS model, the FleSpeech, to generate speaker-aligned emotional audiobook.