Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decoding

Marco Pasini, Stefan Lattner, George Fazekas

TL;DR

Music2Latent2 tackles the trade-off between high compression and audio fidelity by encoding audio into unordered summary embeddings and decoding with a consistency model in an autoregressive, chunked framework. It introduces a two-step decoding procedure and a training strategy on two consecutive chunks with causal masking to ensure coherence across boundaries. Empirically, it achieves substantially better FAD and downstream MIR performance than continuous autoencoders at equivalent and higher compression ratios, with a time compression of $4096\times$ and latent compression of $64\times$. The approach enables scalable, high-fidelity neural audio compression suitable for generative modeling and MIR tasks.

Abstract

Efficiently compressing high-dimensional audio signals into a compact and informative latent space is crucial for various tasks, including generative modeling and music information retrieval (MIR). Existing audio autoencoders, however, often struggle to achieve high compression ratios while preserving audio fidelity and facilitating efficient downstream applications. We introduce Music2Latent2, a novel audio autoencoder that addresses these limitations by leveraging consistency models and a novel approach to representation learning based on unordered latent embeddings, which we call summary embeddings. Unlike conventional methods that encode local audio features into ordered sequences, Music2Latent2 compresses audio signals into sets of summary embeddings, where each embedding can capture distinct global features of the input sample. This enables to achieve higher reconstruction quality at the same compression ratio. To handle arbitrary audio lengths, Music2Latent2 employs an autoregressive consistency model trained on two consecutive audio chunks with causal masking, ensuring coherent reconstruction across segment boundaries. Additionally, we propose a novel two-step decoding procedure that leverages the denoising capabilities of consistency models to further refine the generated audio at no additional cost. Our experiments demonstrate that Music2Latent2 outperforms existing continuous audio autoencoders regarding audio quality and performance on downstream tasks. Music2Latent2 paves the way for new possibilities in audio compression.

Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decoding

TL;DR

Music2Latent2 tackles the trade-off between high compression and audio fidelity by encoding audio into unordered summary embeddings and decoding with a consistency model in an autoregressive, chunked framework. It introduces a two-step decoding procedure and a training strategy on two consecutive chunks with causal masking to ensure coherence across boundaries. Empirically, it achieves substantially better FAD and downstream MIR performance than continuous autoencoders at equivalent and higher compression ratios, with a time compression of and latent compression of . The approach enables scalable, high-fidelity neural audio compression suitable for generative modeling and MIR tasks.

Abstract

Efficiently compressing high-dimensional audio signals into a compact and informative latent space is crucial for various tasks, including generative modeling and music information retrieval (MIR). Existing audio autoencoders, however, often struggle to achieve high compression ratios while preserving audio fidelity and facilitating efficient downstream applications. We introduce Music2Latent2, a novel audio autoencoder that addresses these limitations by leveraging consistency models and a novel approach to representation learning based on unordered latent embeddings, which we call summary embeddings. Unlike conventional methods that encode local audio features into ordered sequences, Music2Latent2 compresses audio signals into sets of summary embeddings, where each embedding can capture distinct global features of the input sample. This enables to achieve higher reconstruction quality at the same compression ratio. To handle arbitrary audio lengths, Music2Latent2 employs an autoregressive consistency model trained on two consecutive audio chunks with causal masking, ensuring coherent reconstruction across segment boundaries. Additionally, we propose a novel two-step decoding procedure that leverages the denoising capabilities of consistency models to further refine the generated audio at no additional cost. Our experiments demonstrate that Music2Latent2 outperforms existing continuous audio autoencoders regarding audio quality and performance on downstream tasks. Music2Latent2 paves the way for new possibilities in audio compression.

Paper Structure

This paper contains 18 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio Autoencoders
  4. Consistency Models
  5. Background
  6. Music2Latent2
  7. Audio Representation
  8. Architecture
  9. Training Process
  10. Inference
  11. Implementation Details
  12. Experiments and Results
  13. Experimental Setting
  14. Influence of $\sigma_{\text{cond}}$
  15. Ablation Study
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Architecture of Music2Latent2. Convolutional patchifiers and de-patchifiers are indicated with P, transformer modules with T. Audio embeddings are illustrated as A, learned/summary embeddings as L, and mask embeddings as M. We represent chunked causal masking with a curved arrow.
  • Figure 2: Autoregressive decoding of Music2Latent2.
  • Figure 3: Impact of $\sigma_{\text{cond}}$ on FAD and $\text{FAD}_\text{clap}$ for two-step decoding.