Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation

Zhengyan Sheng, Zhihao Du, Heng Lu, Shiliang Zhang, Zhen-Hua Ling

TL;DR

This work tackles multimodality-driven speaker generation by enabling voice control from multiple modalities (e.g., face and text) through a unified voice space. It introduces UniSpeaker, which combines a KV-Former–based Multimodal Voice Aggregator with soft contrastive learning and self-distillation on top of the CosyVoice backbone, preserving speech naturalness while enabling cross-modal alignment. The authors also present the MVC benchmark to evaluate face- and text-driven voice control across five tasks, showing UniSpeaker consistently outperforms modality-specific baselines in voice suitability, diversity, and quality. The approach yields a richer, more robust voice space and demonstrates strong robustness to noisy inputs, with clear opportunities for extending to additional modalities.

Abstract

Recent advancements in personalized speech generation have brought synthetic speech increasingly close to the realism of target speakers' recordings, yet multimodal speaker generation remains on the rise. This paper introduces UniSpeaker, a unified approach for multimodality-driven speaker generation. Specifically, we propose a unified voice aggregator based on KV-Former, applying soft contrastive loss to map diverse voice description modalities into a shared voice space, ensuring that the generated voice aligns more closely with the input descriptions. To evaluate multimodality-driven voice control, we build the first multimodality-based voice control (MVC) benchmark, focusing on voice suitability, voice diversity, and speech quality. UniSpeaker is evaluated across five tasks using the MVC benchmark, and the experimental results demonstrate that UniSpeaker outperforms previous modality-specific models. Speech samples are available at \url{https://UniSpeaker.github.io}.

Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation

TL;DR

This work tackles multimodality-driven speaker generation by enabling voice control from multiple modalities (e.g., face and text) through a unified voice space. It introduces UniSpeaker, which combines a KV-Former–based Multimodal Voice Aggregator with soft contrastive learning and self-distillation on top of the CosyVoice backbone, preserving speech naturalness while enabling cross-modal alignment. The authors also present the MVC benchmark to evaluate face- and text-driven voice control across five tasks, showing UniSpeaker consistently outperforms modality-specific baselines in voice suitability, diversity, and quality. The approach yields a richer, more robust voice space and demonstrates strong robustness to noisy inputs, with clear opportunities for extending to additional modalities.

Abstract

Recent advancements in personalized speech generation have brought synthetic speech increasingly close to the realism of target speakers' recordings, yet multimodal speaker generation remains on the rise. This paper introduces UniSpeaker, a unified approach for multimodality-driven speaker generation. Specifically, we propose a unified voice aggregator based on KV-Former, applying soft contrastive loss to map diverse voice description modalities into a shared voice space, ensuring that the generated voice aligns more closely with the input descriptions. To evaluate multimodality-driven voice control, we build the first multimodality-based voice control (MVC) benchmark, focusing on voice suitability, voice diversity, and speech quality. UniSpeaker is evaluated across five tasks using the MVC benchmark, and the experimental results demonstrate that UniSpeaker outperforms previous modality-specific models. Speech samples are available at \url{https://UniSpeaker.github.io}.
Paper Structure (17 sections, 10 equations, 5 figures, 2 tables)

This paper contains 17 sections, 10 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Relate Work
  3. Multimodality-driven voice control for speech generation
  4. Large speech generation models
  5. Methods
  6. Preliminaries
  7. Multimodal Voice Description Integration
  8. Multimodal Voice Aggregator
  9. Soft Contrastive Learning
  10. Self-distillation
  11. Dataset and Benchmark
  12. Experiments
  13. Experiment Settings
  14. Evaluation Results
  15. Ablation Study
  16. ...and 2 more sections

Figures (5)

  • Figure 1: The pipeline of Unispeaker for multiple modalities speaker generation. Initial speaker generation is performed using absolute voice descriptions. If the initial results are unsatisfactory, further voice attributes editing can be done to achieve a finsal speaker generation.
  • Figure 2: The pipeline of Unispeaker for multiple modalities speaker generation. Initial speaker generation is performed using absolute voice descriptions. If the initial results are unsatisfactory, further voice attributes editing can be done to achieve a finsal speaker generation.
  • Figure 3: Average preference scores (%) of ABX tests about voice suitability in comparison, where "N/P" stands for “no preference”.
  • Figure 4: The evaluation results about different multimodal data scales on joint voice modeling
  • Figure 5: The evaluation results about different multimodal data scales on joint voice modeling. Here, the horizontal axis represents the amount of additional multimodal data, with "0" indicating that only the LRS3 dataset was used.