PARASOL: Parametric Style Control for Diffusion Image Synthesis

TL;DR

PARASOL introduces a diffusion-based framework with parametric, disentangled control over image content and fine-grained visual style by jointly conditioning on semantic and style embeddings. A latent diffusion model is trained with modality-specific losses and a projector that maps distinct content/style descriptors into a shared space, enabling cross-attention conditioning and modality-specific classifier-free guidance during sampling. The method is trained on 500k cross-modal triplets obtained via cross-modal search and demonstrates strong style fidelity, robust content preservation, and versatile applications including content/style interpolation and generative visual search. The results indicate substantial improvements over state-of-the-art generative multimodal models and NST alternatives, with practical benefits for creative design and search-driven workflows.

Abstract

We propose PARASOL, a multi-modal synthesis model that enables disentangled, parametric control of the visual style of the image by jointly conditioning synthesis on both content and a fine-grained visual style embedding. We train a latent diffusion model (LDM) using specific losses for each modality and adapt the classifier-free guidance for encouraging disentangled control over independent content and style modalities at inference time. We leverage auxiliary semantic and style-based search to create training triplets for supervision of the LDM, ensuring complementarity of content and style cues. PARASOL shows promise for enabling nuanced control over visual style in diffusion models for image creation and stylization, as well as generative search where text-based search results may be adapted to more closely match user intent by interpolating both content and style descriptors.

Figures (28)

• Figure 2: Comparison to Stable Diffusion ldm. Stable Diffusion encounters difficulty in disentangling content and style as well as transferring the particular requested style, while PARASOL adeptly combines fine-grained details of both into its output.
• Figure 3: Illustration of the full PARASOL pipeline. It consists of six components: A parametric style encoder $A$ (red); A projector network $\mathcal{M}()$ (brown); A semantics encoder $C$ (blue); An Autoencoder $(\mathcal{E}, \mathcal{D})$ (green); A denoising U-Net (orange); An optional post-processing step (purple). At training time (bottom-right corner), two modality-specific losses ($L_s$ and $L_y$) are used to encourage disentanglement. They are combined with $L_{DM}$ and minimized in the training. At inference time (big pipeline), a parameter $\lambda \in [0, T]$ is introduced. After $\lambda$ denoising steps, the style condition is changed for transferring a new style.
• Figure 4: Comparison to Generative Multimodal Models (RDM rdm, ControlNet controlnet, DiffuseIT diffuseit) and PARASOL+.
• Figure 5: Comparison to Style Transfer Models (AdaIN adain, CAST cast, ContrAST contraAST, PAMA pama, SANet sanet, StyTr2 deng2021stytr), PARASOL and PARASOL+.
• Figure 6: Effect of the multimodal loss. Adding the multimodal loss encourages the model to better combine the information from each modality when their descriptors are not fully disentangled.