Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Network Bending of Diffusion Models for Audio-Visual Generation

Luke Dzwonczyk, Carmine Emanuele Cella, David Ban

TL;DR

The paper addresses how to make diffusion-based audio-visual generation artistically expressive with continuous control. It introduces network bending applied to Stable Diffusion, inserting operators into the U-Net and conditioning them on audio features to steer framewise generation, with diffusion timesteps $t$ guiding the evolution from noise to image. A taxonomy of operators (point-wise, tensor, morphological) is developed and demonstrated through two audio-to-video pipelines, yielding effects from color adjustments to scene changes and semantic shifts. The authors provide code and example videos, highlighting a path toward learned, cross-modal control and suggesting avenues for operator learning and extension to other generative models.

Abstract

In this paper we present the first steps towards the creation of a tool which enables artists to create music visualizations using pre-trained, generative, machine learning models. First, we investigate the application of network bending, the process of applying transforms within the layers of a generative network, to image generation diffusion models by utilizing a range of point-wise, tensor-wise, and morphological operators. We identify a number of visual effects that result from various operators, including some that are not easily recreated with standard image editing tools. We find that this process allows for continuous, fine-grain control of image generation which can be helpful for creative applications. Next, we generate music-reactive videos using Stable Diffusion by passing audio features as parameters to network bending operators. Finally, we comment on certain transforms which radically shift the image and the possibilities of learning more about the latent space of Stable Diffusion based on these transforms.

Network Bending of Diffusion Models for Audio-Visual Generation

TL;DR

The paper addresses how to make diffusion-based audio-visual generation artistically expressive with continuous control. It introduces network bending applied to Stable Diffusion, inserting operators into the U-Net and conditioning them on audio features to steer framewise generation, with diffusion timesteps guiding the evolution from noise to image. A taxonomy of operators (point-wise, tensor, morphological) is developed and demonstrated through two audio-to-video pipelines, yielding effects from color adjustments to scene changes and semantic shifts. The authors provide code and example videos, highlighting a path toward learned, cross-modal control and suggesting avenues for operator learning and extension to other generative models.

Abstract

In this paper we present the first steps towards the creation of a tool which enables artists to create music visualizations using pre-trained, generative, machine learning models. First, we investigate the application of network bending, the process of applying transforms within the layers of a generative network, to image generation diffusion models by utilizing a range of point-wise, tensor-wise, and morphological operators. We identify a number of visual effects that result from various operators, including some that are not easily recreated with standard image editing tools. We find that this process allows for continuous, fine-grain control of image generation which can be helpful for creative applications. Next, we generate music-reactive videos using Stable Diffusion by passing audio features as parameters to network bending operators. Finally, we comment on certain transforms which radically shift the image and the possibilities of learning more about the latent space of Stable Diffusion based on these transforms.
Paper Structure (13 sections, 2 equations, 5 figures)

This paper contains 13 sections, 2 equations, 5 figures.

Table of Contents

  1. Introduction
  2. State of the Art
  3. Classical Methods
  4. Learning-based Methods
  5. Methodology
  6. Experiments
  7. Point-wise Operations
  8. Tensor Operations
  9. Morphological Transformations
  10. Audio-to-Video
  11. Discussion
  12. Conclusions and Future Work
  13. Acknowledgments

Figures (5)

  • Figure 1: A block diagram of our system. An operator $f$ is inserted at some layer in the U-Net. The audio feature computed from the audio at that time is passed as the parameter to $f$.
  • Figure 2: Image generations using the prompt "a floating orb" with various point-wise operators applied
  • Figure 3: Image generations using the prompt "a gorgeous landscape" with $R_1$ applied at layer 20 with changing angle
  • Figure 4: Examples of semantic shift. The top image is generated with no transformation applied. The bottom image is the same prompt but with $R_1$ applied at layer 0
  • Figure 5: Examples of scene change with various prompts. The upper image has no transformation applied.