Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models

TL;DR

This work identifies a key weakness in CFG-based fine-tuned diffusion models: degraded unconditional priors undermine conditional generation. It proposes a training-free remedy by substituting the fine-tuned model's unconditional noise with richer unconditional noise from the base or another pretrained diffusion model, which can even be from a model with a different architecture. The approach yields consistent improvements across image and video conditioning tasks, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and InstructPix2Pix, while incurring some memory and speed costs. By framing the sampling as a mixture of unconditional and conditional priors, the method leverages richer priors to sharpen condition alignment and overall quality. The findings highlight practical implications for deploying CFG-based conditionings without retraining, though adapter-based fine-tuning methods may require further investigation.

Abstract

Classifier-Free Guidance (CFG) is a fundamental technique in training conditional diffusion models. The common practice for CFG-based training is to use a single network to learn both conditional and unconditional noise prediction, with a small dropout rate for conditioning. However, we observe that the joint learning of unconditional noise with limited bandwidth in training results in poor priors for the unconditional case. More importantly, these poor unconditional noise predictions become a serious reason for degrading the quality of conditional generation. Inspired by the fact that most CFG-based conditional models are trained by fine-tuning a base model with better unconditional generation, we first show that simply replacing the unconditional noise in CFG with that predicted by the base model can significantly improve conditional generation. Furthermore, we show that a diffusion model other than the one the fine-tuned model was trained on can be used for unconditional noise replacement. We experimentally verify our claim with a range of CFG-based conditional models for both image and video generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and InstructPix2Pix.

Figures (11)

• Figure 1: Unconditional Priors Matter in CFG-Based Conditional Generation. Fine-tuned conditional diffusion models often show drastic degradation in their unconditional priors, adversely affecting conditional generation when using techniques such as CFG ho2021classifier. We demonstrate that leveraging a diffusion model with a richer unconditional prior and combining its unconditional noise prediction with the conditional noise prediction from the fine-tuned model can lead to substantial improvements in conditional generation quality. This is demonstrated across diverse conditional diffusion models including Zero-1-to-3 liu2023zero, Versatile Diffusion xu2023versatile, InstructPix2Pix brooks2023instructpix2pix, and DynamiCrafter xing2025dynamicrafter.
• Figure 2: Unconditional samples from different diffusion models. Stable Diffusion blattmann2023stable, which often serves as the base model for fine-tuning conditional diffusion models, generates plausible images, whereas other fine-tuned diffusion models fail to sample realistic images.
• Figure 3: Novel View Synthesis with Zero-1-to-3 liu2023zero. Outputs from Zero-1-to-3 often show inaccuracies in lighting or shape distortions during novel view synthesis. By incorporating unconditional noise predictions from Stable Diffusion rombach2022high or PixArt-$\alpha$chen2023pixart, our method achieves clear improvements in output quality.
• Figure 4: Image Variations with Versatile Diffusion xu2023versatile. Versatile Diffusion often suffers from style and detail degradation---excessive saturation (rows 1 and 3) or loss of key content (row 2). In contrast, our method, leveraging SD1.4, SD2.1, or PixArt-$\alpha$ as unconditional priors, achieves noticeable improvements in performance.
• Figure 5: Image-to-Video Generation with DynamiCrafter xing2025dynamicrafter. Our method is more temporally consistent (lighting on the biker) and less distorted (the hand and face in the second video).