Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword Spotting

Sichen Jin, Youngmoon Jung, Seungjin Lee, Jaeyoung Roh, Changwoo Han, Hoonyoung Cho

TL;DR

This is the first attempt to dynamically align the audio and the keyword text on-the-fly to attain the joint audio-text embedding for KWS.

Abstract

This paper introduces a novel approach for streaming openvocabulary keyword spotting (KWS) with text-based keyword enrollment. For every input frame, the proposed method finds the optimal alignment ending at the frame using connectionist temporal classification (CTC) and aggregates the frame-level acoustic embedding (AE) to obtain higher-level (i.e., character, word, or phrase) AE that aligns with the text embedding (TE) of the target keyword text. After that, we calculate the similarity of the aggregated AE and the TE. To the best of our knowledge, this is the first attempt to dynamically align the audio and the keyword text on-the-fly to attain the joint audio-text embedding for KWS. Despite operating in a streaming fashion, our approach achieves competitive performance on the LibriPhrase dataset compared to the non-streaming methods with a mere 155K model parameters and a decoding algorithm with time complexity O(U), where U is the length of the target keyword at inference time.

CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword Spotting

TL;DR

This is the first attempt to dynamically align the audio and the keyword text on-the-fly to attain the joint audio-text embedding for KWS.

Abstract

This paper introduces a novel approach for streaming openvocabulary keyword spotting (KWS) with text-based keyword enrollment. For every input frame, the proposed method finds the optimal alignment ending at the frame using connectionist temporal classification (CTC) and aggregates the frame-level acoustic embedding (AE) to obtain higher-level (i.e., character, word, or phrase) AE that aligns with the text embedding (TE) of the target keyword text. After that, we calculate the similarity of the aggregated AE and the TE. To the best of our knowledge, this is the first attempt to dynamically align the audio and the keyword text on-the-fly to attain the joint audio-text embedding for KWS. Despite operating in a streaming fashion, our approach achieves competitive performance on the LibriPhrase dataset compared to the non-streaming methods with a mere 155K model parameters and a decoding algorithm with time complexity O(U), where U is the length of the target keyword at inference time.
Paper Structure (11 sections, 11 equations, 3 figures, 2 tables)

This paper contains 11 sections, 11 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CTC
  4. Multi-view Loss
  5. Proposed Method
  6. Architecture
  7. CTC Aligner
  8. Training Strategy
  9. Inference
  10. Experiment Result
  11. Conclusion

Figures (3)

  • Figure 1: The depiction of the entire structure of the CTC-aligned Audio-Text (CTCAT) keyword detector. The acoustic encoder and the text encoder encodes the input audio sequence and the target keyword tokens into acoustic embedding (AE) and text embedding (TE), and the CTC aligner aligns the embedding vectors that share the same tokens.
  • Figure 2: The decoding graph made for the target keyword "cat". In each state, transition timings and the accumulated AE are stored. The portions colored white in the state means the information is not (half) filled.
  • Figure 3: The correlation map between the audio-text embedding of the positive examples for the target keyword "said the king". The blue lines denote the aligned result from CTC for the audio sections of $\langle silence \rangle , "said","the", "king", \langle silence \rangle$