Ensembling Diffusion Models via Adaptive Feature Aggregation

TL;DR

This work tackles the challenge of improving text-guided image generation by ensembling multiple diffusion models. It introduces Adaptive Feature Aggregation (AFA), featuring a lightweight Spatial-Aware Block-Wise (SABW) aggregator that dynamically fuses block-wise features from several frozen U-Net denoisers, guided by prompts, initial noises, denoising steps, and spatial locations. Through extensive experiments on COCO 2017, Draw Bench Prompts, and additional datasets, AFA consistently outperforms static merging baselines and existing ensembling methods, delivering sharper images and better context alignment, while showing robustness to fewer inference steps. The approach demonstrates strong generality across architectures like SDXL and FLUX with SABW, though it cannot yet mix architectures; ablations confirm the critical role of spatial attention and conditioning signals. Overall, AFA provides a practical, scalable path to leverage multiple high-quality diffusion models to achieve superior and more context-consistent generations.

Abstract

The success of the text-guided diffusion model has inspired the development and release of numerous powerful diffusion models within the open-source community. These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. Leveraging multiple high-quality models to produce stronger generation ability is valuable, but has not been extensively studied. Existing methods primarily adopt parameter merging strategies to produce a new static model. However, they overlook the fact that the divergent denoising capabilities of the models may dynamically change across different states, such as when experiencing different prompts, initial noises, denoising steps, and spatial locations. In this paper, we propose a novel ensembling method, Adaptive Feature Aggregation (AFA), which dynamically adjusts the contributions of multiple models at the feature level according to various states (i.e., prompts, initial noises, denoising steps, and spatial locations), thereby keeping the advantages of multiple diffusion models, while suppressing their disadvantages. Specifically, we design a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptive aggregates the block-wise intermediate features from multiple U-Net denoisers into a unified one. The core idea lies in dynamically producing an individual attention map for each model's features by comprehensively considering various states. It is worth noting that only SABW is trainable with about 50 million parameters, while other models are frozen. Both the quantitative and qualitative experiments demonstrate the effectiveness of our proposed Adaptive Feature Aggregation method.

Figures (9)

• Figure 1: Examples to illustrate the dynamical change of the denoising capabilities across various states. We conduct experiments on different prompts with various initial noises. We then plot the proportion of wins (i.e., the model with the least error between the predicted noise and the initial noise), for each model in a certain spatial region.
• Figure 2: Framework of ensembling multiple diffusion models by our AFA method.
• Figure 3: Quantitative comparison between AFA with the two base models.
• Figure 4: Effect of varying inference steps.
• Figure 5: Qualitative comparison between AFA with the base models and the baselines.