Bootstrapping Diffusion: Diffusion Model Training Leveraging Partial and Corrupted Data
Xudong Ma
TL;DR
Bootstrapping diffusion tackles the data efficiency challenge of diffusion models by leveraging abundant partial-view data (e.g., patches and downsampled images) alongside a small full-resolution dataset. It trains view-specific denoisers for each partial view and combines them to form a target for a residual denoiser, which is regularized by variance to improve generalization. The authors establish generalization bounds linking partial-view information, KL divergence, and model complexity, and validate the approach with experiments on AFHQv2-Cat showing high-quality full-resolution generation from limited data. Overall, the method achieves near first-order data efficiency by aligning residual learning complexity with the amount of information missing from partial views, with potential extensions in distillation and architecture optimization.
Abstract
Training diffusion models requires large datasets. However, acquiring large volumes of high-quality data can be challenging, for example, collecting large numbers of high-resolution images and long videos. On the other hand, there are many complementary data that are usually considered corrupted or partial, such as low-resolution images and short videos. Other examples of corrupted data include videos that contain subtitles, watermarks, and logos. In this study, we investigate the theoretical problem of whether the above partial data can be utilized to train conventional diffusion models. Motivated by our theoretical analysis in this study, we propose a straightforward approach of training diffusion models utilizing partial data views, where we consider each form of complementary data as a view of conventional data. Our proposed approach first trains one separate diffusion model for each individual view, and then trains a model for predicting the residual score function. We prove generalization error bounds, which show that the proposed diffusion model training approach can achieve lower generalization errors if proper regularizations are adopted in the residual score function training. In particular, we prove that the difficulty in training the residual score function scales proportionally with the signal correlations not captured by partial data views. Consequently, the proposed approach achieves near first-order optimal data efficiency.