A Language Model With Million Context Length For Raw Audio
Prateek Verma
TL;DR
The paper tackles the challenge of modeling long-term dependencies in raw audio by introducing an end-to-end generative auto-regressive architecture that first learns a continuous latent representation with a CNN front-end and then models dependencies across these representations using a Transformer. This approach enables very long context lengths (up to hundreds of thousands of samples) while maintaining a compact parameter budget, and it demonstrates state-of-the-art negative-log-likelihood on the YouTube-Mix piano dataset compared to WaveNet, Sample-RNN, and SaSHMI. Key contributions include the use of a continuous latent space rather than discrete codes, a lightweight Transformer over latent tokens, and end-to-end training that jointly optimizes representation learning and next-sample prediction. The results suggest significant potential for scalable long-context audio generation and indicate applicability to broader time-series domains with larger datasets and parameter counts in the future.
Abstract
Modeling long-term dependencies for audio signals is a particularly challenging problem, as even small-time scales yield on the order of a hundred thousand samples. With the recent advent of Transformers, neural architectures became good at modeling dependencies over longer time scales, but they suffered from quadratic constraints to scale them. We propose a generative auto-regressive architecture that can model audio waveforms over quite a large context, greater than 500,000 samples. Our work is adapted to learn time dependencies by learning a latent representation by a CNN front-end, and then learning dependencies over these representations using Transformer encoders, fully trained end-to-end: thereby allowing to learn representations as it deems fit for the next sample. Unlike previous works that compared different time scales to show improvement, we use a standard dataset, with the same number of parameters/context to show improvements. We achieve a state-of-the-art performance as compared to other approaches such as Wavenet, SaSHMI, and Sample-RNN on a standard dataset for modeling long-term structure. This work gives very exciting direction for the field, given improvements in context modeling that can be scaled with more data, as well as potentially better results by using billions/trillions of parameters.