Provable Speech Attributes Conversion via Latent Independence

TL;DR

This work introduces the Independence Conditional Autoencoder (ICAE) as a principled framework for speech attribute conversion with theoretical guarantees. By enforcing independence between a latent content variable and controllable attributes, the model ensures reliable style transformation while preserving content, even under imperfect training. The paper proves that recovering the latent content up to an invertible transformation suffices for correct synthesis and conversion, and it derives a practical error bound that ties reconstruction quality and decoder smoothness to conversion accuracy. The proposed IVC method operationalizes ICAE with a lightweight WaveNet-based architecture, using offline clustering to obtain an approximately independent content proxy and training with two mean-squared-error losses, achieving competitive results on voice and emotion conversion with one-shot inference and linear-time complexity. Together, the theoretical contributions and practical design provide a reproducible baseline for theory-frontier speech attribute manipulation and lay groundwork for extending to multimodal settings.

Abstract

While signal conversion and disentangled representation learning have shown promise for manipulating data attributes across domains such as audio, image, and multimodal generation, existing approaches, especially for speech style conversion, are largely empirical and lack rigorous theoretical foundations to guarantee reliable and interpretable control. In this work, we propose a general framework for speech attribute conversion, accompanied by theoretical analysis and guarantees under reasonable assumptions. Our framework builds on a non-probabilistic autoencoder architecture with an independence constraint between the predicted latent variable and the target controllable variable. This design ensures a consistent signal transformation, conditioned on an observed style variable, while preserving the original content and modifying the desired attribute. We further demonstrate the versatility of our method by evaluating it on speech styles, including speaker identity and emotion. Quantitative evaluations confirm the effectiveness and generality of the proposed approach.

Figures (4)

• Figure 1: An autoencoder is trained to minimize a reconstruction loss $\mathcal{R}(\mathbf{X}, \hat{\mathbf{X}})$ and an independence loss $\mathcal{I}(\mathbf{C}, \hat{\mathbf{S}})$, ensuring that the latent $\hat{\mathbf{S}}$ is statistically independent of the condition $\mathbf{C}$ while accurately reconstructing $\mathbf{X}$.
• Figure 2: The proposed IVC framework for speech attributes conversion. First, $K$-means clustering of speech features $\mathbf{X}$ is trained offline to obtain $\mathcal{P}_{\tilde{\mathbf{S}} \mid \mathbf{C}} \approx \mathcal{P}_{\tilde{\mathbf{S}}}$. Then the model is trained to extract $\hat{\mathbf{S}} \approx \tilde{\mathbf{S}}$ and $\mathbf{X} \approx \hat{\mathbf{X}}$.
• Figure 3: We validate Assumption \ref{['assmp:nonuniform']} by computing the pairwise distance matrix $\mathbf{D}$ over the prior probabilities of the speech unit categories $\tilde{s}_k$, for $k \in \{1,\dots,K\}$. Since $\tilde{\mathbf{S}}$ serves as a proxy for $\mathbf{S}$ and $\mathbf{D}_{ii} = 0 \;\; \forall i, \quad \mathbf{D}_{ij} \neq 0 \;\; \forall i \neq j$, this analysis provides evidence that the assumption is satisfied in real-world speech data.
• Figure 4: Distributions of cosine distances calculated on the pairs constructed from real samples (orange) and on the pairs constructed such that sample one comes from generated samples and sample two comes from a real sample (blue). The distance is defined by $1-\frac{\mathbf{x}_i \cdot \mathbf{x}_j}{||\mathbf{x}_i||_2||\mathbf{x}_j||_2}$ where $\mathbf{x}_i$, $\mathbf{x}_j$ are speaker embeddings. The results obtained from the model trained with the RedimNet speaker encoder. The plot shows that the generated samples are very close to the real samples and almost indistinguishable.

Theorems & Definitions (12)

• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof
• Proposition 1
• proof
• Corollary 1
• proof