Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play

TL;DR

Voila introduces a family of voice-language foundation models that unify semantic and acoustic processing for real-time, autonomous voice interactions. Through a hierarchical multi-scale Transformer, a Voila tokenizer that produces discrete audio tokens, and a text-audio interleaved alignment strategy, Voila delivers low-latency, full-duplex communication and supports extensive voice customization with over a million pre-built voices. The approach achieves competitive ASR and TTS performance while enabling proactive, persona-aware dialogue and seamless integration across tasks like translation. With a publicly released codebase and models, Voila aims to accelerate research and the development of next-generation autonomous voice agents with practical impact in daily human-machine interactions.

Abstract

A voice AI agent that blends seamlessly into daily life would interact with humans in an autonomous, real-time, and emotionally expressive manner. Rather than merely reacting to commands, it would continuously listen, reason, and respond proactively, fostering fluid, dynamic, and emotionally resonant interactions. We introduce Voila, a family of large voice-language foundation models that make a step towards this vision. Voila moves beyond traditional pipeline systems by adopting a new end-to-end architecture that enables full-duplex, low-latency conversations while preserving rich vocal nuances such as tone, rhythm, and emotion. It achieves a response latency of just 195 milliseconds, surpassing the average human response time. Its hierarchical multi-scale Transformer integrates the reasoning capabilities of large language models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware voice generation -- where users can simply write text instructions to define the speaker's identity, tone, and other characteristics. Moreover, Voila supports over one million pre-built voices and efficient customization of new ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue, Voila is designed as a unified model for a wide range of voice-based applications, including automatic speech recognition (ASR), Text-to-Speech (TTS), and, with minimal adaptation, multilingual speech translation. Voila is fully open-sourced to support open research and accelerate progress toward next-generation human-machine interactions.

Figures (7)

• Figure 1: Different paradigms of voice conversation systems: (a) Traditional pipeline systems, such as Apple Siri, Amazon Alexa, and Google Assistant, launched in the 2010s; (b) Simplified pipeline systems using LLMs to handle text-based understanding and response generation; (c) End-to-end audio-in, audio-out systems that offer low latency and rich vocal nuances; (d) Autonomous systems that further enable dynamic, proactive interactions.
• Figure 2: Voila models: (a)Voila-e2e for end-to-end voice conversation, (b)Voila-autonomous for autonomous interaction. Both models allow easy customization of speaker characteristics and voice via text instructions and audio samples.
• Figure 3: Text and audio interleaved alignment.
• Figure 4: Input embedding and output decoding in Voila.
• Figure 5: Voila-autonomous two-stream inputs, including user’s audio stream and Voila’s own audio stream.