Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Promptformer: Prompted Conformer Transducer for ASR

Sergio Duarte-Torres, Arunasish Sen, Aman Rana, Lukas Drude, Alejandro Gomez-Alanis, Andreas Schwarz, Leif Rädel, Volker Leutnant

TL;DR

This work tackles exploiting cross-utterance textual context in streaming ASR by fusing text and acoustic representations within a Conformer-Transducer encoder via a prompt-based mechanism inspired by hyper-prompting. It introduces a single global prompt that augments acoustic keys/values, using either an external BERT-based or an internal embedding-based context generator, with token-level SP-based representations providing the best trade-off when combined with copying initialization. Empirical results on a large in-house English voice assistant dataset show up to 5.9% relative WER reduction in multi-turn scenarios, while maintaining robustness when context is absent and reducing computational overhead. The approach remains effective with limited fine-tuning, notably by updating only the prompt and projection layers, indicating practical benefits for latency-constrained, context-aware ASR systems.

Abstract

Context cues carry information which can improve multi-turn interactions in automatic speech recognition (ASR) systems. In this paper, we introduce a novel mechanism inspired by hyper-prompting to fuse textual context with acoustic representations in the attention mechanism. Results on a test set with multi-turn interactions show that our method achieves 5.9% relative word error rate reduction (rWERR) over a strong baseline. We show that our method does not degrade in the absence of context and leads to improvements even if the model is trained without context. We further show that leveraging a pre-trained sentence-piece model for context embedding generation can outperform an external BERT model.

Promptformer: Prompted Conformer Transducer for ASR

TL;DR

This work tackles exploiting cross-utterance textual context in streaming ASR by fusing text and acoustic representations within a Conformer-Transducer encoder via a prompt-based mechanism inspired by hyper-prompting. It introduces a single global prompt that augments acoustic keys/values, using either an external BERT-based or an internal embedding-based context generator, with token-level SP-based representations providing the best trade-off when combined with copying initialization. Empirical results on a large in-house English voice assistant dataset show up to 5.9% relative WER reduction in multi-turn scenarios, while maintaining robustness when context is absent and reducing computational overhead. The approach remains effective with limited fine-tuning, notably by updating only the prompt and projection layers, indicating practical benefits for latency-constrained, context-aware ASR systems.

Abstract

Context cues carry information which can improve multi-turn interactions in automatic speech recognition (ASR) systems. In this paper, we introduce a novel mechanism inspired by hyper-prompting to fuse textual context with acoustic representations in the attention mechanism. Results on a test set with multi-turn interactions show that our method achieves 5.9% relative word error rate reduction (rWERR) over a strong baseline. We show that our method does not degrade in the absence of context and leads to improvements even if the model is trained without context. We further show that leveraging a pre-trained sentence-piece model for context embedding generation can outperform an external BERT model.
Paper Structure (13 sections, 4 equations, 1 figure, 3 tables)

This paper contains 13 sections, 4 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. METHOD
  3. Streaming Conformer Transducer
  4. Context Generation
  5. Context Consumption
  6. Experimental Setup
  7. Model configuration
  8. Datasets
  9. Experimental results
  10. Context integration approaches
  11. Textual Context Representation
  12. Training regime
  13. Conclusions

Figures (1)

  • Figure 1: Proposed system for generating and consuming textual context prompts. Dotted lines represent the various ablative configurations described in our result section.