Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Factorized Neural Transducer Model For text-only Domain Adaptation

Junzhe Liu, Jianwei Yu, Xie Chen

TL;DR

This work tackles the challenge of text-only domain adaptation for end-to-end ASR by proposing the Improved Factorized Neural Transducer (IFNT), which tightly integrates acoustic and language information and enables effective text-based adaptation. IFNT extends the factorized approach by applying a sigmoid to the vocabulary decoder, projecting the LM output to the joint space, and directly incorporating the LM posterior over the vocabulary into the final logits, while training end-to-end from scratch with the same loss form as FNT. Across English and Mandarin, IFNT yields consistent improvements in baseline accuracy over both NT and FNT and demonstrates superior adaptation, achieving up to $30.2\%$ relative WER reductions on out-of-domain data compared with the standard NT with shallow fusion and up to $2.8\%$ relative gains over FNT in several settings. These results suggest that early fusion of acoustic and linguistic information within a unified Transducer framework, together with text-only LM adaptation, can significantly enhance robustness to domain mismatch in scalable ASR systems.

Abstract

Adapting End-to-End ASR models to out-of-domain datasets with text data is challenging. Factorized neural Transducer (FNT) aims to address this issue by introducing a separate vocabulary decoder to predict the vocabulary. Nonetheless, this approach has limitations in fusing acoustic and language information seamlessly. Moreover, a degradation in word error rate (WER) on the general test sets was also observed, leading to doubts about its overall performance. In response to this challenge, we present the improved factorized neural Transducer (IFNT) model structure designed to comprehensively integrate acoustic and language information while enabling effective text adaptation. We assess the performance of our proposed method on English and Mandarin datasets. The results indicate that IFNT not only surpasses the neural Transducer and FNT in baseline performance in both scenarios but also exhibits superior adaptation ability compared to FNT. On source domains, IFNT demonstrated statistically significant accuracy improvements, achieving a relative enhancement of 1.2% to 2.8% in baseline accuracy compared to the neural Transducer. On out-of-domain datasets, IFNT shows relative WER(CER) improvements of up to 30.2% over the standard neural Transducer with shallow fusion, and relative WER(CER) reductions ranging from 1.1% to 2.8% on test sets compared to the FNT model.

Improved Factorized Neural Transducer Model For text-only Domain Adaptation

TL;DR

This work tackles the challenge of text-only domain adaptation for end-to-end ASR by proposing the Improved Factorized Neural Transducer (IFNT), which tightly integrates acoustic and language information and enables effective text-based adaptation. IFNT extends the factorized approach by applying a sigmoid to the vocabulary decoder, projecting the LM output to the joint space, and directly incorporating the LM posterior over the vocabulary into the final logits, while training end-to-end from scratch with the same loss form as FNT. Across English and Mandarin, IFNT yields consistent improvements in baseline accuracy over both NT and FNT and demonstrates superior adaptation, achieving up to relative WER reductions on out-of-domain data compared with the standard NT with shallow fusion and up to relative gains over FNT in several settings. These results suggest that early fusion of acoustic and linguistic information within a unified Transducer framework, together with text-only LM adaptation, can significantly enhance robustness to domain mismatch in scalable ASR systems.

Abstract

Adapting End-to-End ASR models to out-of-domain datasets with text data is challenging. Factorized neural Transducer (FNT) aims to address this issue by introducing a separate vocabulary decoder to predict the vocabulary. Nonetheless, this approach has limitations in fusing acoustic and language information seamlessly. Moreover, a degradation in word error rate (WER) on the general test sets was also observed, leading to doubts about its overall performance. In response to this challenge, we present the improved factorized neural Transducer (IFNT) model structure designed to comprehensively integrate acoustic and language information while enabling effective text adaptation. We assess the performance of our proposed method on English and Mandarin datasets. The results indicate that IFNT not only surpasses the neural Transducer and FNT in baseline performance in both scenarios but also exhibits superior adaptation ability compared to FNT. On source domains, IFNT demonstrated statistically significant accuracy improvements, achieving a relative enhancement of 1.2% to 2.8% in baseline accuracy compared to the neural Transducer. On out-of-domain datasets, IFNT shows relative WER(CER) improvements of up to 30.2% over the standard neural Transducer with shallow fusion, and relative WER(CER) reductions ranging from 1.1% to 2.8% on test sets compared to the FNT model.
Paper Structure (15 sections, 4 equations, 1 figure, 4 tables)

This paper contains 15 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Standard and Factorized Neural Transducer Models
  3. Standard neural Transducer
  4. LM shallow fusion
  5. Factorized Neural Transducer
  6. Improved factorized neural Transducer
  7. Experiments
  8. Datasets
  9. Experiment setups
  10. In-domain evaluation
  11. Out-of-domain evaluation
  12. Ablation study and analysis
  13. Limitations
  14. Conclusion
  15. Acknowledgements

Figures (1)

  • Figure 1: The illustration of three model structures: (a) standard neural Transducer (NT). (b) factorized neural Transducer (FNT). (c) proposed improved factorized neural Transducer (IFNT).