Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models

Heting Gao, Junrui Ni, Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson

TL;DR

WavPrompt investigates transferring few-shot learning to audio by freezing a language model and learning an audio encoder that produces embeddings compatible with the LM. The system uses a wav2vec 2.0-based encoder and GPT-2, plus a downsampling layer and ASR pretraining, to answer speech-based prompts without fine-tuning the LM. Calibrated prompting and extensive ablations identify a concrete configuration that yields consistent gains over a naïve ASR+LM baseline across several speech tasks and even extends to non-speech audio. The work demonstrates practical potential for end-to-end few-shot audio understanding and provides actionable guidelines for designing audio-to-language adapters.

Abstract

Large-scale auto-regressive language models pretrained on massive text have demonstrated their impressive ability to perform new natural language tasks with only a few text examples, without the need for fine-tuning. Recent studies further show that such a few-shot learning ability can be extended to the text-image setting by training an encoder to encode the images into embeddings functioning like the text embeddings of the language model. Interested in exploring the possibility of transferring the few-shot learning ability to the audio-text setting, we propose a novel speech understanding framework, WavPrompt, where we finetune a wav2vec model to generate a sequence of audio embeddings understood by the language model. We show that WavPrompt is a few-shot learner that can perform speech understanding tasks better than a naive text baseline. We conduct detailed ablation studies on different components and hyperparameters to empirically identify the best model configuration. In addition, we conduct a non-speech understanding experiment to show WavPrompt can extract more information than just the transcriptions. Code is available at https://github.com/Hertin/WavPrompt

WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models

TL;DR

WavPrompt investigates transferring few-shot learning to audio by freezing a language model and learning an audio encoder that produces embeddings compatible with the LM. The system uses a wav2vec 2.0-based encoder and GPT-2, plus a downsampling layer and ASR pretraining, to answer speech-based prompts without fine-tuning the LM. Calibrated prompting and extensive ablations identify a concrete configuration that yields consistent gains over a naïve ASR+LM baseline across several speech tasks and even extends to non-speech audio. The work demonstrates practical potential for end-to-end few-shot audio understanding and provides actionable guidelines for designing audio-to-language adapters.

Abstract

Large-scale auto-regressive language models pretrained on massive text have demonstrated their impressive ability to perform new natural language tasks with only a few text examples, without the need for fine-tuning. Recent studies further show that such a few-shot learning ability can be extended to the text-image setting by training an encoder to encode the images into embeddings functioning like the text embeddings of the language model. Interested in exploring the possibility of transferring the few-shot learning ability to the audio-text setting, we propose a novel speech understanding framework, WavPrompt, where we finetune a wav2vec model to generate a sequence of audio embeddings understood by the language model. We show that WavPrompt is a few-shot learner that can perform speech understanding tasks better than a naive text baseline. We conduct detailed ablation studies on different components and hyperparameters to empirically identify the best model configuration. In addition, we conduct a non-speech understanding experiment to show WavPrompt can extract more information than just the transcriptions. Code is available at https://github.com/Hertin/WavPrompt
Paper Structure (15 sections, 2 equations, 3 figures, 3 tables)

This paper contains 15 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Model Architecture
  4. Downsampling Layer
  5. Speech Recognition Pretraining
  6. Few-Shot Evaluation
  7. Calibration
  8. Experiments
  9. Datasets
  10. Experiment Setup and Baseline
  11. Results on the Speech Understanding Tasks
  12. Ablation Studies
  13. Generalizing to Non-Speech Tasks
  14. Conclusions
  15. Acknowledgements

Figures (3)

  • Figure 1: Interface of WavPrompt
  • Figure 2: Results of speech understanding tasks.
  • Figure 3: Classification accuracy versus number of shots. Shaded region is $\pm 1$ standard deviations.