SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks

TL;DR

The paper investigates prompting textless speech language models by casting diverse speech processing tasks as speech to unit generation using discrete quantized units. It compares encoder–decoder and decoder–only unit LMs, and introduces input and deep prompt tuning, fixed and learnable verbalizers, and a unified pipeline that handles speech classification, sequence generation, and speech generation. Experimental results show prompting can achieve competitive performance with significantly fewer trainable parameters than fine tuning, with Unit mBART excelling in generation tasks and learnable verbalizers providing interpretability and gains in several settings. The work highlights the potential of prompt driven speech LMs for efficient, scalable, and privacy-friendly speech processing, while outlining limitations and avenues for future research as speech LMs become more advanced.

Abstract

Prompting has become a practical method for utilizing pre-trained language models (LMs). This approach offers several advantages. It allows an LM to adapt to new tasks with minimal training and parameter updates, thus achieving efficiency in both storage and computation. Additionally, prompting modifies only the LM's inputs and harnesses the generative capabilities of language models to address various downstream tasks in a unified manner. This significantly reduces the need for human labor in designing task-specific models. These advantages become even more evident as the number of tasks served by the LM scales up. Motivated by the strengths of prompting, we are the first to explore the potential of prompting speech LMs in the domain of speech processing. Recently, there has been a growing interest in converting speech into discrete units for language modeling. Our pioneer research demonstrates that these quantized speech units are highly versatile within our unified prompting framework. Not only can they serve as class labels, but they also contain rich phonetic information that can be re-synthesized back into speech signals for speech generation tasks. Specifically, we reformulate speech processing tasks into speech-to-unit generation tasks. As a result, we can seamlessly integrate tasks such as speech classification, sequence generation, and speech generation within a single, unified prompting framework. The experiment results show that the prompting method can achieve competitive performance compared to the strong fine-tuning method based on self-supervised learning models with a similar number of trainable parameters. The prompting method also shows promising results in the few-shot setting. Moreover, with the advanced speech LMs coming into the stage, the proposed prompting framework attains great potential.

Figures (6)

• Figure 1: Comparison of the "pre-train, fine-tune" paradigm with the prompting paradigm. The "pre-train, fine-tune" paradigm involves designing task-specific downstream models and loss functions by human experts, with distinct models trained for each task. In contrast, the prompting paradigm handles all downstream tasks in a unified manner, where only the prompt varies for each task, while the language model remains fixed.
• Figure 2: The textless speech LM. It consists of three components, including (1) The speech-to-unit encoder, (2) the unit language model, and (3) the unit-to-speech decoder.
• Figure 3: An overview of the proposed framework, where all downstream tasks are treated as speech-to-unit generation processes. The generation of units is directed by the task-specific prompts that guide the unit language model. A verbalizer or speech decoder then bridges the gap between the generated units and the corresponding downstream labels.
• Figure 4: An overview of the proposed framework, where all downstream tasks are treated as speech-to-unit generation processes. The generation of units is directed by the task-specific prompts that guide the unit language model. A verbalizer or speech decoder then bridges the gap between the generated units and the corresponding downstream labels.
• Figure 5: Illustration of the learnable verbalizer. The logits are transformed into labels for the downstream task through a linear transformation. Furthermore, the original vocabulary embeddings are converted into class-specific embeddings using weighted transformations, aligning them more closely with the downstream task.