ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing

TL;DR

ArtCrafter tackles the challenge of balancing expressive textual semantics with stylistic fidelity and output diversity in diffusion-based text-to-image style transfer. It introduces three modules—Attention-based Style Extraction (ASE), Text-Image Aligning Augmentation (TIAA), and Explicit Modulation (EM)—along with an embedding reframing framework and a dedicated ArtMarket dataset to align image and text embeddings in a shared space. Across qualitative, quantitative, and human studies, ArtCrafter achieves stronger text–style alignment, richer artistic details, and greater output diversity than prior methods, while remaining compatible with existing controllable tools and adaptable to natural and long-form prompts. The approach promises practical impact for creative AI workflows, enabling nuanced, diverse, and controllable style transfer with efficient, lightweight training.

Abstract

Recent years have witnessed significant advancements in text-guided style transfer, primarily attributed to innovations in diffusion models. These models excel in conditional guidance, utilizing text or images to direct the sampling process. However, despite their capabilities, direct conditional guidance approaches often face challenges in balancing the expressiveness of textual semantics with the diversity of output results while capturing stylistic features. To address these challenges, we introduce ArtCrafter, a novel framework for text-to-image style transfer. Specifically, we introduce an attention-based style extraction module, meticulously engineered to capture the subtle stylistic elements within an image. This module features a multi-layer architecture that leverages the capabilities of perceiver attention mechanisms to integrate fine-grained information. Additionally, we present a novel text-image aligning augmentation component that adeptly balances control over both modalities, enabling the model to efficiently map image and text embeddings into a shared feature space. We achieve this through attention operations that enable smooth information flow between modalities. Lastly, we incorporate an explicit modulation that seamlessly blends multimodal enhanced embeddings with original embeddings through an embedding reframing design, empowering the model to generate diverse outputs. Extensive experiments demonstrate that ArtCrafter yields impressive results in visual stylization, exhibiting exceptional levels of stylistic intensity, controllability, and diversity.

Figures (17)

• Figure 1: ArtCrafter generation results. By injecting the features of the reference images and text prompts during the diffusion process, our method is capable of capturing and generating a faithful style representation.
• Figure 2: Generic adapter-based vs. ArtCrafter generation results. Given a content description of "Fashion Shoes", generic adapter-based generation (above) results in unaligned results and limited result diversity. In contrast, our approach (below) generates text-aligned content as well as multiple shoe types.
• Figure 3: Our proposed ArtCrafter's overview pipeline consists of three modules: 1) Attention-based style extraction captures fine-grained style features from images using multi-head perceiver attention and feed-forward networks. 2) Text-image aligning augmentation transfers the style embedding to the textual space to improve the image-text consistency in the generated images. 3) Explicit modulation seamlessly combines multimodal embeddings with original image embeddings and text embeddings in different ways, increasing the versatility and diversity of the method.
• Figure 4: Qualitative Comparison with Other State-of-the-Art Text-Guided Stylization Methods. We conducted a comprehensive qualitative evaluation by comparing our method with various state-of-the-art text-guided stylization methods, including Styleshot styleshot, Style Aligned stylealigned, VSP vsp, InstantStyle instantstyle, InstantStyle (Plus) instantstyle, IP-Adapter ipadapter, IP-Adapter (SDXL) ipadapter, CSGO csgo, and StyleCrafter liu2023stylecrafter.
• Figure 5: Ablation studies for attention-based Style Extraction. (a) Visual differences between ArtCrafter generated samples without preceiver attention and our method. (b) Discussion on the number of learnable queries N.