GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization

TL;DR

GimmBO addresses the challenge of interactive, high-dimensional adapter merging for diffusion models by framing merging as a continuous design problem guided by human preferences. It introduces a two-stage preferential Bayesian optimization backend that enforces sparsity via a $B$-capped simplex and then polishes the active set, enabling efficient exploration of 20–30 adapters. A Gaussian process surrogate with a SAAS prior learns from pairwise user rankings, while content control via SDEdit helps separate style from content. Through simulated user experiments and a user study, GimmBO demonstrates faster convergence, higher success rates, and better sparsity recovery than line-search and prior BO baselines, and shows extensibility to content merging, coarse retrieval integration, and downstream stylization pipelines. The work provides a practical framework for interactive, preference-driven design in large adapter spaces with broad implications for personalized image generation and retrieval-augmented workflows.

Abstract

Fine-tuning-based adaptation is widely used to customize diffusion-based image generation, leading to large collections of community-created adapters that capture diverse subjects and styles. Adapters derived from the same base model can be merged with weights, enabling the synthesis of new visual results within a vast and continuous design space. To explore this space, current workflows rely on manual slider-based tuning, an approach that scales poorly and makes weight selection difficult, even when the candidate set is limited to 20-30 adapters. We propose GimmBO to support interactive exploration of adapter merging for image generation through Preferential Bayesian Optimization (PBO). Motivated by observations from real-world usage, including sparsity and constrained weight ranges, we introduce a two-stage BO backend that improves sampling efficiency and convergence in high-dimensional spaces. We evaluate our approach with simulated users and a user study, demonstrating improved convergence, high success rates, and consistent gains over BO and line-search baselines, and further show the flexibility of the framework through several extensions.

Figures (17)

• Figure 1: The adapter design space is vast. GimmBO assists users to produce substantially different yet equally satisfying results.
• Figure 2: Previous BO-based interaction methods koyama2020sequential, which do not account for the structure of the adapter-merging design space, may propose clamped or degraded samples in high-dimensional settings.
• Figure 3: Starting from initial samples, GimmBO iteratively proposes new sample batches (pink stars and images), guided by an acquisition function (blue) over a surrogate model $\hat{f}$ updated with accumulated samples (blue dots and images). In this 2D example, GimmBO rapidly converges to the target.
• Figure 4: Batch samples at Steps 1 and 10 in a 20D matching task ($q=8$). Blue cells denote non-zero coefficients (darker = larger magnitude), white cells indicate zeros; green arrows mark ground-truth active axes. With identical initialization and random seed, $B=2$ performs best compared to $B=1$ and $B=n$.
• Figure 5: Running-best similarity curves averaged over 30 prompt-weight combinations and 5 random seeds, together with success rates and sparsity recovery (F1; black horizontal lines indicate the median) of the final best result. Success rates are first reported overall and then broken down by the ground-truth number of active adapters $z$, measured at a similarity threshold of $0.9$ (solid bars), with darker outlines indicating cases exceeding $0.95$. Curves are shown for two individual test inputs, with images corresponding to the highlighted iterations (orange circles).