Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VoxMorph: Scalable Zero-shot Voice Identity Morphing via Disentangled Embeddings

Bharath Krishnamurthy, Ajita Rattani

TL;DR

VoxMorph introduces a zero-shot voice identity morphing framework that disentangles prosody and timbre into separate embeddings and uses Slerp to blend them, enabling high-fidelity morphs from as little as 5 seconds of input without retraining. The fused embeddings condition a three-stage generator: an autoregressive LM produces acoustic tokens guided by the prosody embedding, a Conditional Flow Matching model generates a mel-spectrogram conditioned on the timbre embedding, and a HiFTNet vocoder yields the final waveform. Empirical results show state-of-the-art audio quality, intelligibility, and morphing efficacy under strict security thresholds, with VoxMorph-v2 achieving the best performance across metrics such as FAD, KLD, WER, MMPMR, and FMMPMR. The work underscores a scalable, practical threat to ASV systems and provides a public dataset of 10,000 morphs to spur defense research, while highlighting directions for multi-identity, cross-lingual, and real-time synthesis.

Abstract

Morphing techniques generate artificial biometric samples that combine features from multiple individuals, allowing each contributor to be verified against a single enrolled template. While extensively studied in face recognition, this vulnerability remains largely unexplored in voice biometrics. Prior work on voice morphing is computationally expensive, non-scalable, and limited to acoustically similar identity pairs, constraining practical deployment. Moreover, existing sound-morphing methods target audio textures, music, or environmental sounds and are not transferable to voice identity manipulation. We propose VoxMorph, a zero-shot framework that produces high-fidelity voice morphs from as little as five seconds of audio per subject without model retraining. Our method disentangles vocal traits into prosody and timbre embeddings, enabling fine-grained interpolation of speaking style and identity. These embeddings are fused via Spherical Linear Interpolation (Slerp) and synthesized using an autoregressive language model coupled with a Conditional Flow Matching network. VoxMorph achieves state-of-the-art performance, delivering a 2.6x gain in audio quality, a 73% reduction in intelligibility errors, and a 67.8% morphing attack success rate on automated speaker verification systems under strict security thresholds. This work establishes a practical and scalable paradigm for voice morphing with significant implications for biometric security. The code and dataset are available on our project page: https://vcbsl.github.io/VoxMorph/

VoxMorph: Scalable Zero-shot Voice Identity Morphing via Disentangled Embeddings

TL;DR

VoxMorph introduces a zero-shot voice identity morphing framework that disentangles prosody and timbre into separate embeddings and uses Slerp to blend them, enabling high-fidelity morphs from as little as 5 seconds of input without retraining. The fused embeddings condition a three-stage generator: an autoregressive LM produces acoustic tokens guided by the prosody embedding, a Conditional Flow Matching model generates a mel-spectrogram conditioned on the timbre embedding, and a HiFTNet vocoder yields the final waveform. Empirical results show state-of-the-art audio quality, intelligibility, and morphing efficacy under strict security thresholds, with VoxMorph-v2 achieving the best performance across metrics such as FAD, KLD, WER, MMPMR, and FMMPMR. The work underscores a scalable, practical threat to ASV systems and provides a public dataset of 10,000 morphs to spur defense research, while highlighting directions for multi-identity, cross-lingual, and real-time synthesis.

Abstract

Morphing techniques generate artificial biometric samples that combine features from multiple individuals, allowing each contributor to be verified against a single enrolled template. While extensively studied in face recognition, this vulnerability remains largely unexplored in voice biometrics. Prior work on voice morphing is computationally expensive, non-scalable, and limited to acoustically similar identity pairs, constraining practical deployment. Moreover, existing sound-morphing methods target audio textures, music, or environmental sounds and are not transferable to voice identity manipulation. We propose VoxMorph, a zero-shot framework that produces high-fidelity voice morphs from as little as five seconds of audio per subject without model retraining. Our method disentangles vocal traits into prosody and timbre embeddings, enabling fine-grained interpolation of speaking style and identity. These embeddings are fused via Spherical Linear Interpolation (Slerp) and synthesized using an autoregressive language model coupled with a Conditional Flow Matching network. VoxMorph achieves state-of-the-art performance, delivering a 2.6x gain in audio quality, a 73% reduction in intelligibility errors, and a 67.8% morphing attack success rate on automated speaker verification systems under strict security thresholds. This work establishes a practical and scalable paradigm for voice morphing with significant implications for biometric security. The code and dataset are available on our project page: https://vcbsl.github.io/VoxMorph/
Paper Structure (14 sections, 1 equation, 1 figure, 3 tables)

This paper contains 14 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Proposed Framework
  4. Disentangled Vocal Feature Extraction
  5. Interpolation
  6. Multi-Stage Synthesis of the Morphed Voice
  7. Experimental Setup
  8. Datasets and Implementation
  9. Baselines
  10. Evaluation Metrics and Implementation Details
  11. Results and Analysis
  12. Quantitative Comparison with Baselines
  13. Ablation Study
  14. Conclusion

Figures (1)

  • Figure 1: Architectural overview of the VoxMorph framework. The process consists of three core stages: (1) Extraction: Disentangled prosody (style) and timbre (identity) embeddings are extracted from two speaker identities. (2) Interpolation: The embedding representations are independently interpolated using Slerp. (3) Synthesis/ Morphing: The fused prosody embedding conditions an autoregressive language model, while the fused timbre embedding guides a flow matching network to generate a mel-spectrogram. A neural vocoder then converts the spectrogram into a high-fidelity morphed waveform.