Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects

TL;DR

This work addresses the limited speech technology support for Chinese dialects by pursuing cross-dialect semantic alignment between dialects and Mandarin using ASR-only data. It introduces YuBao, a comprehensive Chinese-dialect speech benchmark, and develops a Zipformer-based dialect ASR model that covers major Sinitic subgroups. The authors demonstrate that ASR data alone can yield semantically aligned speech representations, as evidenced by zero-shot speech-to-speech retrieval across dialects, with strong recall against Mandarin and dialect pairs. The approach enables dialect-to-Mandarin speech-LLMs and lays groundwork for future dialect-focused speech technologies, including broader Benchmark expansion and model distillation techniques.

Abstract

Despite having hundreds of millions of speakers, Chinese dialects lag behind Mandarin in speech and language technologies. Most varieties are primarily spoken, making dialect-to-Mandarin speech-LLMs (large language models) more practical than dialect LLMs. Building dialect-to-Mandarin speech-LLMs requires speech representations with cross-dialect semantic alignment between Chinese dialects and Mandarin. In this paper, we achieve such a cross-dialect semantic alignment by training a speech encoder with ASR (automatic speech recognition)-only data, as demonstrated by speech-to-speech retrieval on a new benchmark of spoken Chinese varieties that we contribute. Our speech encoder further demonstrates state-of-the-art ASR performance on Chinese dialects. Together, our Chinese dialect benchmark, semantically aligned speech representations, and speech-to-speech retrieval evaluation lay the groundwork for future Chinese dialect speech-LLMs. We release the benchmark at https://github.com/kalvinchang/yubao.

Figures (3)

• Figure 1: Architecture of our models trained with ASR-only or ASR + speech translation (ST) data (Sec. \ref{['sec:asr-model']}). During training, ASR data goes through both the RNN-T head and the attention head ("decoder"), while ST goes through the attention head ("decoder") only. During inference, the RNN-T head is used for ASR, while the attention head ("decoder") can be used for ST. The gray boxes illustrate the speech encoder embeddings used in our retrieval experiments.
• Figure 2: Illustration of speech-to-speech retrieval between a pair of dialect sites. Suppose there is a spoken corpus composed of 8 sentences with the same meaning across different dialects. Then the goal is to measure how well the speech embeddings can match the utterance in a source dialect to the utterance in the target dialect with the same meaning. The matching (retrieval) is done by identifying the target dialect utterance with the highest embedding similarity (Fig. \ref{['fig:seq-sim']}) to the source dialect utterance. Embedding similarity (Fig. \ref{['fig:seq-sim']}) is computed for each each cell in this figure, which represents one pair of sentences between a source and target dialect. A retrieved pair is identified as correct if both utterances have the same meaning, i.e. they lie along the diagonal in this figure.
• Figure 3: Illustration of how SeqSim ma-etal-2025-cross between a pair of Chinese dialect sites in YuBao is computed. The cosine similarity between all pairs of speech encoder frames between a source utterance's embeddings and a target utterance's embeddings is calculated. Then the maximum similarity is taken across each row and column to obtain the final SeqSim score according to \ref{['eq:seqsim']}.