FreeU: Free Lunch in Diffusion U-Net

TL;DR

This work reveals that diffusion U-Nets denoise primarily via their backbone while skip connections inject high-frequency details that can hamper backbone-driven denoising. It proposes FreeU, an inference-time reweighting scheme that separately scales backbone and skip features, incorporating a structure-aware backbone scaling and Fourier-domain modulation for skip features, all without additional training. FreeU consistently improves image and video generation quality across multiple diffusion foundations (e.g., Stable Diffusion, DreamBooth, ModelScope, ReVersion, Rerender) and downstream tasks, as demonstrated by qualitative results and human evaluations. The approach is lightweight, easily integrated, and broadly applicable to diffusion-based generation workflows, offering practical gains with negligible overhead.

Abstract

In this paper, we uncover the untapped potential of diffusion U-Net, which serves as a "free lunch" that substantially improves the generation quality on the fly. We initially investigate the key contributions of the U-Net architecture to the denoising process and identify that its main backbone primarily contributes to denoising, whereas its skip connections mainly introduce high-frequency features into the decoder module, causing the network to overlook the backbone semantics. Capitalizing on this discovery, we propose a simple yet effective method-termed "FreeU" - that enhances generation quality without additional training or finetuning. Our key insight is to strategically re-weight the contributions sourced from the U-Net's skip connections and backbone feature maps, to leverage the strengths of both components of the U-Net architecture. Promising results on image and video generation tasks demonstrate that our FreeU can be readily integrated to existing diffusion models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion, to improve the generation quality with only a few lines of code. All you need is to adjust two scaling factors during inference. Project page: https://chenyangsi.top/FreeU/.

Figures (18)

• Figure 1: We propose FreeU, a method that substantially improves diffusion model sample quality at no costs: no training, no additional parameter introduced, and no increase in memory or sampling time.
• Figure 2: The denoising process. The top row illustrates the image's progressive denoising process across iterations, while the subsequent two rows display low-frequency and high-frequency components after the inverse Fourier Transform, matching each step. It's evident that low-frequency components change slowly, whereas high-frequency components exhibit more significant variations during the denoising process.
• Figure 3: Relative log amplitudes of Fourier with variations of the backbone scaling factor $b$. Increasing in $b$ correspondingly results in a suppression of high-frequency components in the images generated by the diffusion model.
• Figure 4: FreeU Framework. (a) U-Net Skip Features and Backbone Features. In U-Net, the skip features and backbone features are concatenated together at each decoding stage. We apply the FreeU operations during concatenation. (b) FreeU Operations. The factor $b$ aims to amplify the backbone feature map $\boldsymbol{x}$, while factor $s$ is designed to attenuate the skip feature map $\boldsymbol{h}$.
• Figure 5: Effect of backbone and skip connection scaling factors ($b$ and $s$). Increasing the backbone scaling factor $b$ significantly enhances image quality, while variations in the skip scaling factor $s$ have a negligible influence on image synthesis quality.