Toward a Sparse and Interpretable Audio Codec

TL;DR

The work tackles the interpretability gap in block-based audio codecs by introducing a sparse, event-based encoder paired with a physics-inspired decoder. The system represents audio as a sequence of time-stamped events with compact 32-dimensional vectors, operating on STFT magnitude inputs and enabling streaming playback and manipulation. Trained on MusicNet with an iterative, energy-reduction loss, the approach yields highly sparse representations and offers intuitive, inspectable rendering steps through resonances and room responses, though perceptual fidelity lags behind traditional codecs. If refined with perceptually-informed losses and alternative architectures, this framework could provide musically meaningful compression and a versatile substrate for downstream generative or interactive applications.

Abstract

Most widely-used modern audio codecs, such as Ogg Vorbis and MP3, as well as more recent "neural" codecs like Meta's Encodec or the Descript Audio Codec are based on block-coding; audio is divided into overlapping, fixed-size "frames" which are then compressed. While they often yield excellent reproductions and can be used for downstream tasks such as text-to-audio, they do not produce an intuitive, directly-interpretable representation. In this work, we introduce a proof-of-concept audio encoder that represents audio as a sparse set of events and their times-of-occurrence. Rudimentary physics-based assumptions are used to model attack and the physical resonance of both the instrument being played and the room in which a performance occurs, hopefully encouraging a sparse, parsimonious, and easy-to-interpret representation.

Figures (5)

• Figure 1: In this visualization of the codec representation, we see that events can overlap and vary in length. Time is along the x-axis, event positions are along the y-axis and event colors are chosen by applying t-SNEvanDerMaaten2008 to the set of 32 event vectors, targeting a single dimension for the y-axis and three dimensions to represent an RGB color value.
• Figure 2: High-level model architecture. The encoder and decoder work together to incrementally remove energy from the input representation, an STFT-based magnitude spectrogram. At each step, the encoder produces a single event vector and time-of-occurrence. The decoder generates an audio event, positions it in time, and subtracts it from the input representation.
• Figure 3: Here, we can see the anatomy of a decoder block, which performs something like source-excitation synthesis. A burst of noise is convolved with a number of decaying resonances. A time-varying mixture interpolates between the different resonances over time, after which the original impulse and the resonant signal are mixed together before being output.
• Figure 4: This scatterplot shows event vectors from a large number of audio segments mapped onto a 2D-plane using t-SNEvanDerMaaten2008. Exploring nearby neighbors can locate variations of a query event. An interactive version of this scatterplot can be explored https://blog.cochlea.xyz/scatter.html.
• Figure 5: Here we can see intermediate stages of the source-excitation-based decoder at work. From left to right, we see A. a spectrogram of the initial noisy impulse B. a spectogram of the noisy impulse convolved with the chosen decaying resonance and C. the result from step B. convolved with a room impulse response.