FlexIO: Flexible Single- and Multi-Channel Speech Separation and Enhancement

TL;DR

FlexIO tackles the challenge of universal speech separation and enhancement by enabling a single model to process variable numbers of microphones $M$ and speakers $N$ via prompt-driven conditioning and an array-agnostic channel communication mechanism. Building on the single-channel TUSS backbone, FlexIO introduces a multi-channel cross-prompt module that exchanges information across channels (via TAC, cross-channel attention, or co-attention) and applies conditional TSE at a reference channel using $N$ prompts to control outputs. Empirically, FlexIO demonstrates robust performance across 1–5 microphones and 1–3 speakers, generalizes to unseen 3- and 5-channel conditions, and achieves competitive perceptual and recognition metrics on CHiME-4 compared with baselines like USES and U2-C. The work offers a flexible, scalable path toward universal SSE and points to promising extensions to general audio source separation and joint source counting in future work.

Abstract

Speech separation and enhancement (SSE) has advanced remarkably and achieved promising results in controlled settings, such as a fixed number of speakers and a fixed array configuration. Towards a universal SSE system, single-channel systems have been extended to deal with a variable number of speakers (i.e., outputs). Meanwhile, multi-channel systems accommodating various array configurations (i.e., inputs) have been developed. However, these attempts have been pursued separately. In this paper, we propose a flexible input and output SSE system, named FlexIO. It performs conditional separation using prompt vectors, one per speaker as a condition, allowing separation of an arbitrary number of speakers. Multi-channel mixtures are processed together with the prompt vectors via an array-agnostic channel communication mechanism. Our experiments demonstrate that FlexIO successfully covers diverse conditions with one to five microphones and one to three speakers. We also confirm the robustness of FlexIO on CHiME-4 real data.

Figures (1)

• Figure 1: FlexIO takes $N$ prompt vectors $\mathbf{p}$ and an $M$-channel mixture $(\mathbf{X}_1, \ldots, \mathbf{X}_M)$ as input, and isolates each speaker at a reference channel $\underline{m}$. Modules of the same color share parameters.