Exploration of Adapter for Noise Robust Automatic Speech Recognition
Hao Shi, Tatsuya Kawahara
TL;DR
This work investigates adapter-based transfer learning to improve automatic speech recognition (ASR) in unseen noisy environments. It systematically studies where to insert adapters in a Conformer-based backend, how the adapter's embedding dimension influences performance, and how training data type (real vs. simulated) and multi-condition training affect adaptation, including integration with a speech enhancement (SE) front-end. Key findings show that placing adapters in shallow encoder layers provides the strongest gains, the embedding dimension is largely robust, and real data generally outperform simulated data at the same amounts; combining adapters with SE front-ends yields further improvements. The results offer practical guidelines for deploying adapters to achieve noise-robust ASR with limited data and point to future work in designing adapters tailored for robustness in noisy conditions.
Abstract
Adapting an automatic speech recognition (ASR) system to unseen noise environments is crucial. Integrating adapters into neural networks has emerged as a potent technique for transfer learning. This study thoroughly investigates adapter-based ASR adaptation in noisy environments. We conducted experiments using the CHiME--4 dataset. The results show that inserting the adapter in the shallow layer yields superior effectiveness, and there is no significant difference between adapting solely within the shallow layer and adapting across all layers. The simulated data helps the system to improve its performance under real noise conditions. Nonetheless, when the amount of data is the same, the real data is more effective than the simulated data. Multi-condition training is still useful for adapter training. Furthermore, integrating adapters into speech enhancement-based ASR systems yields substantial improvements.