Audible Networks: Deconstructing and Manipulating Sounds with Deep Non-Negative Autoencoders

TL;DR

This paper introduces Non-Negative Autoencoders (NAEs) as interpretable, unsupervised tools for deconstructing and creatively manipulating sounds, extending Non-Negative Matrix Factorization (NMF) into deep, hierarchical architectures. By enforcing non-negativity through projected gradient descent, NAEs yield interpretable spectral bases and temporal envelopes, enabling component listening and cross-component or cross-layer synthesis for novel sound designs. The authors demonstrate deep, hierarchical deconstructions, discuss resynthesis via Wiener masks to conserve the original content, and propose randomization and cross-layer operations to controllably alter timbre and event density. They also address training stability, introduce sparsity regularization for deeper networks, and outline future work including stereo/multi-channel extensions and probabilistic formulations, positioning NAEs as a bridge between factorization-based interpretability and deep generative capabilities for object-based sound editing.

Abstract

We propose the use of Non-Negative Autoencoders (NAEs) for sound deconstruction and user-guided manipulation of sounds for creative purposes. NAEs offer a versatile and scalable extension of traditional Non-Negative Matrix Factorization (NMF)-based approaches for interpretable audio decomposition. By enforcing non-negativity constraints through projected gradient descent, we obtain decompositions where internal weights and activations can be directly interpreted as spectral shapes and temporal envelopes, and where components can themselves be listened to as individual sound events. In particular, multi-layer Deep NAE architectures enable hierarchical representations with an adjustable level of granularity, allowing sounds to be deconstructed at multiple levels of abstraction: from high-level note envelopes down to fine-grained spectral details. This framework enables a wide new range of expressive, controllable, and randomized sound transformations. We introduce novel manipulation operations including cross-component and cross-layer synthesis, hierarchical deconstructions, and several randomization strategies that control timbre and event density. Through visualizations and resynthesis of practical examples, we demonstrate how NAEs can serve as flexible and interpretable tools for object-based sound editing.

Figures (4)

• Figure 1: Conceptual representation of a Non-negative Autoencoder (NAE) for sound analysis. An input spectrogram (left) is fed to the network. The weights ($\mathbf{W}$) and activations ($\mathbf{H}$) of the bottleneck layer can be interpreted as a set of, respectively, temporal and spectral elements of the input sound.
• Figure 2: Visualization of a 2-layer NAE deconstruction of a simple mixture of 3 sources.
• Figure 3: Visualization of a 3-layer sparse DNAE deconstruction of a simple mixture of 3 sources.
• Figure 4: Examples of hierarchical deconstructions of Fig. \ref{['fig:nae_2_layer']}. Silent weight vectors are greyed out.