SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages

TL;DR

This work tackles learning speech representations that are robust to speaker variation while preserving lexical tone information in low-resource tonal languages. It introduces SITA, a two-stage adaptation of pretrained wav2vec-style encoders: Stage 1 uses cross-gender contrastive learning and explicit tone repulsion to sculpt a speaker-invariant, tone-aware embedding space, while Stage 2 fine-tunes for ASR with a CTC objective and optional knowledge distillation to maintain recognition performance. Evaluations on Hmong show improved cross-gender word retrieval and reduced tone collapse, with Stage 2 preserving recognition structure; transfer experiments on Mandarin Tone Perfect indicate the method generalizes to other tonal languages. The results suggest SITA is a practical, lightweight plug-in strategy for adapting multilingual speech encoders to tonal languages, offering tangible gains in representation quality and robustness with modest ASR trade-offs.

Abstract

Tonal low-resource languages are widely spoken yet remain underserved by modern speech technology. A key challenge is learning representations that are robust to nuisance variation such as gender while remaining tone-aware for different lexical meanings. To address this, we propose SITA, a lightweight adaptation recipe that enforces Speaker-Invariance and Tone-Awareness for pretrained wav2vec-style encoders. SITA uses staged multi-objective training: (i) a cross-gender contrastive objective encourages lexical consistency across speakers, while a tone-repulsive loss prevents tone collapse by explicitly separating same-word different-tone realizations; and (ii) an auxiliary Connectionist Temporal Classification (CTC)-based ASR objective with distillation stabilizes recognition-relevant structure. We evaluate primarily on Hmong, a highly tonal and severely under-resourced language where off-the-shelf multilingual encoders fail to represent tone effectively. On a curated Hmong word corpus, SITA improves cross-gender lexical retrieval accuracy, while maintaining usable ASR accuracy relative to an ASR-adapted XLS-R teacher. We further observe similar gains when transferring the same recipe to Mandarin, suggesting SITA is a general, plug-in approach for adapting multilingual speech encoders to tonal languages.

Figures (6)

• Figure 1: Two failure modes of speech embedding collapse: (1) demographic bias, where embeddings of the same word spoken by different speakers are insufficiently similar; and (2) not tone-sensitive, where embeddings of the same base word carrying different tones are not meaningfully distinguishable.
• Figure 2: SITA in Two Stages. Stage 1: Speaker-invariant and Tone-sensitive Representation Learning. Stage 2: CTC Fine-tuning with Knowledge Distillation.
• Figure 3: Trade-off between demographic robustness and tone geometry on Hmong. We plot Avg. Top-1 cross-gender retrieval against (a) positive similarity (same word, same tone) and (b) hard-negative cosine distance (same word, different tone). Baselines exhibit tone collapse while SITA achieves strong retrieval with substantial cross-tone separation.
• Figure 4: We visualize token embeddings using PCA (2D) computed from $\ell_2$-normalized pooled encoder representations. Points denote word tokens; colors are tones and marker shapes are base words. Dashed lines connect tone variants within each base word. Whisper shows tone collapse (within-word tones overlap), whereas SITA separates tone variants into a more tone-stratified geometry, aligning with improved hard-negative separation and retrieval.
• Figure 5: Tone perturbation robustness on Hmong word embeddings. For each model, we report the mean cosine similarity between the original word and versions with pitch shifts of $-2$, $-1$, $-0.5$, $+0.5$, $+1$, and $+2$ semitones. All four variants maintain high similarity under mild shifts (e.g., $\pm 0.5$), indicating robustness to small acoustic fluctuations, while similarity drops as the pitch shift magnitude increases, especially for the larger $\pm 2$ semitone changes. The 21-layer models show a slightly steeper decay than the 19-layer models, suggesting stronger sensitivity to large tone perturbations while remaining stable under realistic within-speaker variation.