Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Music2Latent: Consistency Autoencoders for Latent Audio Compression

Marco Pasini, Stefan Lattner, George Fazekas

TL;DR

Music2Latent addresses the challenge of efficiently compressing high-dimensional audio into a continuous latent space suitable for generative modeling and MIR tasks. It introduces a fully end-to-end consistency autoencoder that enables high-fidelity single-step reconstruction, supported by a 4096x time compression and a novel conditioning strategy across UNet layers. Key innovations include frequency-wise self-attention and a learned frequency scaling mechanism to balance frequency components across noise levels, resulting in superior reconstruction quality and competitive MIR performance. The work demonstrates, for the first time in audio, a successful end-to-end consistency autoencoder with strong practical implications for fast decoding and latent-based downstream tasks.

Abstract

Efficient audio representations in a compressed continuous latent space are critical for generative audio modeling and Music Information Retrieval (MIR) tasks. However, some existing audio autoencoders have limitations, such as multi-stage training procedures, slow iterative sampling, or low reconstruction quality. We introduce Music2Latent, an audio autoencoder that overcomes these limitations by leveraging consistency models. Music2Latent encodes samples into a compressed continuous latent space in a single end-to-end training process while enabling high-fidelity single-step reconstruction. Key innovations include conditioning the consistency model on upsampled encoder outputs at all levels through cross connections, using frequency-wise self-attention to capture long-range frequency dependencies, and employing frequency-wise learned scaling to handle varying value distributions across frequencies at different noise levels. We demonstrate that Music2Latent outperforms existing continuous audio autoencoders in sound quality and reconstruction accuracy while achieving competitive performance on downstream MIR tasks using its latent representations. To our knowledge, this represents the first successful attempt at training an end-to-end consistency autoencoder model.

Music2Latent: Consistency Autoencoders for Latent Audio Compression

TL;DR

Music2Latent addresses the challenge of efficiently compressing high-dimensional audio into a continuous latent space suitable for generative modeling and MIR tasks. It introduces a fully end-to-end consistency autoencoder that enables high-fidelity single-step reconstruction, supported by a 4096x time compression and a novel conditioning strategy across UNet layers. Key innovations include frequency-wise self-attention and a learned frequency scaling mechanism to balance frequency components across noise levels, resulting in superior reconstruction quality and competitive MIR performance. The work demonstrates, for the first time in audio, a successful end-to-end consistency autoencoder with strong practical implications for fast decoding and latent-based downstream tasks.

Abstract

Efficient audio representations in a compressed continuous latent space are critical for generative audio modeling and Music Information Retrieval (MIR) tasks. However, some existing audio autoencoders have limitations, such as multi-stage training procedures, slow iterative sampling, or low reconstruction quality. We introduce Music2Latent, an audio autoencoder that overcomes these limitations by leveraging consistency models. Music2Latent encodes samples into a compressed continuous latent space in a single end-to-end training process while enabling high-fidelity single-step reconstruction. Key innovations include conditioning the consistency model on upsampled encoder outputs at all levels through cross connections, using frequency-wise self-attention to capture long-range frequency dependencies, and employing frequency-wise learned scaling to handle varying value distributions across frequencies at different noise levels. We demonstrate that Music2Latent outperforms existing continuous audio autoencoders in sound quality and reconstruction accuracy while achieving competitive performance on downstream MIR tasks using its latent representations. To our knowledge, this represents the first successful attempt at training an end-to-end consistency autoencoder model.
Paper Structure (20 sections, 14 equations, 3 figures, 3 tables)

This paper contains 20 sections, 14 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Autoencoders for Latent Generative Modeling
  4. Consistency Models
  5. Background
  6. Consistency Models
  7. Consistency Training
  8. Music2Latent
  9. Audio Representation
  10. Architecture
  11. Training Process
  12. Implementation Details
  13. Experiments and Results
  14. Datasets
  15. Baselines
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Training process of Music2Latent. The input sample is first encoded into a sequence of latent vectors. The latents are then upsampled with a decoder model. The consistency model is trained via consistency training, with an additional information leakage coming from the cross connections.
  • Figure 2: Audio quality of reconstructed samples with respect to the number of denoising steps of the consistency model.
  • Figure :