Whisper-GPT: A Hybrid Representation Audio Large Language Model

TL;DR

WHISPER-GPT introduces a hybrid audio LLM that jointly leverages continuous mel-spectrogram inputs and discrete ENCODEC tokens within a decoder-only Transformer. By incorporating a Whisper-inspired continuous path alongside discrete token prediction, the model addresses the context-length challenges of purely token-based audio generation while enabling sampling advantages from the discrete space. Empirical results show the hybrid approach achieving superior or competitive next-token NLL and perplexity compared with token-only baselines, with notable gains in music, and achieving similar performance with a fraction of the parameters of larger discrete-token models. This work advances efficient, scalable speech and music generation by enabling effective integration of continuous and discrete audio representations in LLM frameworks.

Abstract

We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture. There has been a huge surge in generative audio, speech, and music models that utilize discrete audio tokens derived from neural compression algorithms, e.g. ENCODEC. However, one of the major drawbacks of this approach is handling the context length. It blows up for high-fidelity generative architecture if one has to account for all the audio contents at various frequencies for the next token prediction. By combining continuous audio representation like the spectrogram and discrete acoustic tokens, we retain the best of both worlds: Have all the information needed from the audio at a specific time instance in a single token, yet allow LLM to predict the future token to allow for sampling and other benefits discrete space provides. We show how our architecture improves the perplexity and negative log-likelihood scores for the next token prediction compared to a token-based LLM for speech and music.

Figures (3)

• Figure 1: (Left) Whisper Architecture proposed by OpenAI radford2023robust which treats ASR as a sequence to sequence which takes in mel-spectrogram slices and decodes it token by token. It has a Transformer Encoder stack on the spectrogram followed by a Transformer decoder, trained for the shift-by-one token prediction, and the cross-attention module on learned spectrogram representation. (Right) Our generative model combines both continuous and discrete representations. We align the spectrogram and ENCODEC coarse tokens. Instead of a Transformer encoder, we pass spectrogram slices through lightweight decoder blocks. The learned representation per-token slice is concatenated with discrete tokens corresponding to the spectrogram slice to have a decoder Transformer stack, trained on shift by one next token prediction, similar to a typical LLM pre-training.
• Figure 2: Comparison of GPT on coarse acoustic tokens with i) GPT-L ii) Our hybrid continuous-discrete representation.
• Figure 3: Comparison of GPT on coarse acoustic tokens with i) GPT-L ii) Our hybrid continuous-discrete representation.