Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognition

Hung-Yang Sung, Chien-Chun Wang, Kuan-Tang Huang, Tien-Hong Lo, Yu-Sheng Tsao, Yung-Chang Hsu, Berlin Chen

TL;DR

This work tackles the challenge of automatic speech recognition for a low-resource language, Taiwanese Hokkien, by proposing CLiFT-ASR, a cross-lingual two-stage fine-tuning framework that starts from Mandarin HuBERT representations. It first learns acoustic/phonetic cues from Tai-lo annotations and then acquires vocabulary and syntax from Han-character transcriptions, enabling effective alignment between speech sounds and orthographic structures. On the TAT-MOE corpus, CLiFT-ASR achieves a 24.88% relative reduction in character error rate (CER) compared with strong baselines, demonstrating a parameter-efficient approach that leverages cross-lingual knowledge. The method shows potential to extend to other low-resource languages and highlights the value of staged, multimodal annotations for improving end-to-end ASR in mixed-orthography settings.

Abstract

Automatic speech recognition (ASR) for low-resource languages such as Taiwanese Hokkien is difficult due to the scarcity of annotated data. However, direct fine-tuning on Han-character transcriptions often fails to capture detailed phonetic and tonal cues, while training only on romanization lacks lexical and syntactic coverage. In addition, prior studies have rarely explored staged strategies that integrate both annotation types. To address this gap, we present CLiFT-ASR, a cross-lingual fine-tuning framework that builds on Mandarin HuBERT models and progressively adapts them to Taiwanese Hokkien. The framework employs a two-stage process in which it first learns acoustic and tonal representations from phonetic Tai-lo annotations and then captures vocabulary and syntax from Han-character transcriptions. This progressive adaptation enables effective alignment between speech sounds and orthographic structures. Experiments on the TAT-MOE corpus demonstrate that CLiFT-ASR achieves a 24.88\% relative reduction in character error rate (CER) compared with strong baselines. The results indicate that CLiFT-ASR provides an effective and parameter-efficient solution for Taiwanese Hokkien ASR and that it has potential to benefit other low-resource language scenarios.

CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognition

TL;DR

This work tackles the challenge of automatic speech recognition for a low-resource language, Taiwanese Hokkien, by proposing CLiFT-ASR, a cross-lingual two-stage fine-tuning framework that starts from Mandarin HuBERT representations. It first learns acoustic/phonetic cues from Tai-lo annotations and then acquires vocabulary and syntax from Han-character transcriptions, enabling effective alignment between speech sounds and orthographic structures. On the TAT-MOE corpus, CLiFT-ASR achieves a 24.88% relative reduction in character error rate (CER) compared with strong baselines, demonstrating a parameter-efficient approach that leverages cross-lingual knowledge. The method shows potential to extend to other low-resource languages and highlights the value of staged, multimodal annotations for improving end-to-end ASR in mixed-orthography settings.

Abstract

Automatic speech recognition (ASR) for low-resource languages such as Taiwanese Hokkien is difficult due to the scarcity of annotated data. However, direct fine-tuning on Han-character transcriptions often fails to capture detailed phonetic and tonal cues, while training only on romanization lacks lexical and syntactic coverage. In addition, prior studies have rarely explored staged strategies that integrate both annotation types. To address this gap, we present CLiFT-ASR, a cross-lingual fine-tuning framework that builds on Mandarin HuBERT models and progressively adapts them to Taiwanese Hokkien. The framework employs a two-stage process in which it first learns acoustic and tonal representations from phonetic Tai-lo annotations and then captures vocabulary and syntax from Han-character transcriptions. This progressive adaptation enables effective alignment between speech sounds and orthographic structures. Experiments on the TAT-MOE corpus demonstrate that CLiFT-ASR achieves a 24.88\% relative reduction in character error rate (CER) compared with strong baselines. The results indicate that CLiFT-ASR provides an effective and parameter-efficient solution for Taiwanese Hokkien ASR and that it has potential to benefit other low-resource language scenarios.

Paper Structure

This paper contains 20 sections, 7 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Linguistic Characteristics of Taiwanese Hokkien
  4. Orthographic Systems and Their Role in ASR
  5. Proposed Method
  6. Model Architecture
  7. Training Strategy
  8. Experimental Setup
  9. Dataset
  10. Data preprocessing
  11. Model Configuration
  12. Training Details
  13. Evaluation Metric
  14. Results and Discussion
  15. Effects of Language Initialization
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of the proposed CLiFT-ASR. The $\bigoplus$ operator denotes element-wise tensor addition. The dashed arrow indicates that during inference, ground-truth labels are not available, so the model outputs are fed back autoregressively into the prediction network.
  • Figure 2: Row-normalized substitution confusion matrices for Taiwanese Hokkien tone prediction, comparing the baseline and proposed models. Tone labels are derived from Taibun surface forms without applying sandhi rules.