Covo-Audio Technical Report

TL;DR

Covo-Audio tackles the challenge of unified audio-language intelligence by introducing a compact 7B end-to-end LALM that processes continuous audio and generates audio within a single model. It combines a Whisper-based audio encoder, a WavLM-derived speech tokenizer, a Flow-Matching decoder, and a BigVGAN vocoder, all trained via a two-stage, large-scale pretraining that fuses continuous acoustic features, discrete tokens, and natural language text through Hierarchical Tri-modal Speech-Text Interleaving. The architecture is augmented with an intelligence-speaker decoupling strategy and a native full-duplex variant (Covo-Audio-Chat-FD) that supports robust, low-latency dialogue with barge-ins and backchanneling. Across pre-training, spoken dialogue benchmarks, and audio understanding tasks, Covo-Audio achieves competitive or state-of-the-art performance for models of comparable scale, demonstrating the viability of 7B-scale models for high-fidelity audio intelligence and scalable voice-enabled AI applications.

Abstract

In this work, we present Covo-Audio, a 7B-parameter end-to-end LALM that directly processes continuous audio inputs and generates audio outputs within a single unified architecture. Through large-scale curated pretraining and targeted post-training, Covo-Audio achieves state-of-the-art or competitive performance among models of comparable scale across a broad spectrum of tasks, including speech-text modeling, spoken dialogue, speech understanding, audio understanding, and full-duplex voice interaction. Extensive evaluations demonstrate that the pretrained foundation model exhibits strong speech-text comprehension and semantic reasoning capabilities on multiple benchmarks, outperforming representative open-source models of comparable scale. Furthermore, Covo-Audio-Chat, the dialogue-oriented variant, demonstrates strong spoken conversational abilities, including understanding, contextual reasoning, instruction following, and generating contextually appropriate and empathetic responses, validating its applicability to real-world conversational assistant scenarios. Covo-Audio-Chat-FD, the evolved full-duplex model, achieves substantially superior performance on both spoken dialogue capabilities and full-duplex interaction behaviors, demonstrating its competence in practical robustness. To mitigate the high cost of deploying end-to-end LALMs for natural conversational systems, we propose an intelligence-speaker decoupling strategy that separates dialogue intelligence from voice rendering, enabling flexible voice customization with minimal text-to-speech (TTS) data while preserving dialogue performance. Overall, our results highlight the strong potential of 7B-scale models to integrate sophisticated audio intelligence with high-level semantic reasoning, and suggest a scalable path toward more capable and versatile LALMs.

Figures (4)

• Figure 1: An Overview of Comprehensive Performance Comparison.
• Figure 2: An Overview of Covo-Audio.
• Figure 3: Data Utilization in Intelligence-Speaker Decoupling Technique.
• Figure 4: A Comparative Overview of Covo-Audio-Chat and Covo-Audio-Chat-FD.