Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M-CIF: Multi-Scale Alignment For CIF-Based Non-Autoregressive ASR

Ruixiang Mao, Xiangnan Ma, Qing Yang, Ziming Zhu, Yucheng Qiao, Yuan Ge, Tong Xiao, Shengxiang Gao, Zhengtao Yu, Jingbo Zhu

TL;DR

This work addresses instability of Continuous Integrate-and-Fire (CIF) based non-autoregressive ASR on synthetic languages by introducing Multi-scale CIF (M-CIF), which progressively aligns character- and phoneme-level information into word-level representations using scale-specific CTC supervision. The method enriches CIF with a hierarchical three-level framework (character, phoneme, word) and a curriculum of losses, enabling more robust acoustic–text alignment and reducing phonetic confusions (PE) and segmentation errors (SE). Empirical results show consistent WER reductions on LibriSpeech and CommonVoice (German and French), while revealing limited gains on Chinese, highlighting language-dependent benefits of multi-scale supervision. The paper further analyzes PE/SE and text-timestamp alignment to demonstrate how the finer-grained guidance improves alignment fidelity and boundary recognition, suggesting practical improvements for NAR ASR in multi-syllabic languages.

Abstract

The Continuous Integrate-and-Fire (CIF) mechanism provides effective alignment for non-autoregressive (NAR) speech recognition. This mechanism creates a smooth and monotonic mapping from acoustic features to target tokens, achieving performance on Mandarin competitive with other NAR approaches. However, without finer-grained guidance, its stability degrades in some languages such as English and French. In this paper, we propose Multi-scale CIF (M-CIF), which performs multi-level alignment by integrating character and phoneme level supervision progressively distilled into subword representations, thereby enhancing robust acoustic-text alignment. Experiments show that M-CIF reduces WER compared to the Paraformer baseline, especially on CommonVoice by 4.21% in German and 3.05% in French. To further investigate these gains, we define phonetic confusion errors (PE) and space-related segmentation errors (SE) as evaluation metrics. Analysis of these metrics across different M-CIF settings reveals that the phoneme and character layers are essential for enhancing progressive CIF alignment.

M-CIF: Multi-Scale Alignment For CIF-Based Non-Autoregressive ASR

TL;DR

This work addresses instability of Continuous Integrate-and-Fire (CIF) based non-autoregressive ASR on synthetic languages by introducing Multi-scale CIF (M-CIF), which progressively aligns character- and phoneme-level information into word-level representations using scale-specific CTC supervision. The method enriches CIF with a hierarchical three-level framework (character, phoneme, word) and a curriculum of losses, enabling more robust acoustic–text alignment and reducing phonetic confusions (PE) and segmentation errors (SE). Empirical results show consistent WER reductions on LibriSpeech and CommonVoice (German and French), while revealing limited gains on Chinese, highlighting language-dependent benefits of multi-scale supervision. The paper further analyzes PE/SE and text-timestamp alignment to demonstrate how the finer-grained guidance improves alignment fidelity and boundary recognition, suggesting practical improvements for NAR ASR in multi-syllabic languages.

Abstract

The Continuous Integrate-and-Fire (CIF) mechanism provides effective alignment for non-autoregressive (NAR) speech recognition. This mechanism creates a smooth and monotonic mapping from acoustic features to target tokens, achieving performance on Mandarin competitive with other NAR approaches. However, without finer-grained guidance, its stability degrades in some languages such as English and French. In this paper, we propose Multi-scale CIF (M-CIF), which performs multi-level alignment by integrating character and phoneme level supervision progressively distilled into subword representations, thereby enhancing robust acoustic-text alignment. Experiments show that M-CIF reduces WER compared to the Paraformer baseline, especially on CommonVoice by 4.21% in German and 3.05% in French. To further investigate these gains, we define phonetic confusion errors (PE) and space-related segmentation errors (SE) as evaluation metrics. Analysis of these metrics across different M-CIF settings reveals that the phoneme and character layers are essential for enhancing progressive CIF alignment.
Paper Structure (12 sections, 7 equations, 4 figures, 2 tables)

This paper contains 12 sections, 7 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. METHOD
  3. CIF Firing Analysis
  4. Multi-scale CIF Strategy
  5. Experiments
  6. Data and Settings
  7. Overall Performance
  8. Analysis
  9. Ablation Study
  10. PE and SE Metrics Analysis
  11. CIF Text-timestamp Alignment Analysis
  12. CONCLUSION

Figures (4)

  • Figure 1: Visualization of text-timestamp alignment for CIF and human annotations on a Chinese–English case. Blue and orange spans show CIF activations and human references; red text marks recognition errors; bottom blue peaks denote accumulated CIF weights.
  • Figure 2: In the CIF activation process, the feed-forward network predicts $\alpha_{\text{pre}}$; the threshold $\beta$ is set to 1, with $\beta_{\text{tail}}$ set to 0.45.
  • Figure 3: Method overview. (a) Progressive integration of character-level and phoneme-level features into word-level representations, aligned with scale-matched CTC; (b) In Paraformer, M-CIF replaces the base CIF, serving as a fine-grained bridge between the encoder and decoder.
  • Figure 4: Comparison of text–timestamp alignments between different M-CIF settings and the human annotations.