StyleBooth: Image Style Editing with Multimodal Instruction

TL;DR

StyleBooth addresses multimodal instruction-based image editing by unifying text and exemplar inputs into a common conditioning space for diffusion models, using <style> and <image> tokens. It couples a trainable mapping W with a scale weighting mechanism to support text-, exemplar-, and compositional style editing, and builds a high-quality dataset through Iterative Style-Destyle Tuning with CLIP-based filtering. The approach achieves strong performance on text-based and exemplar-based editing tasks, including competitive quantitative metrics and improved content preservation, and enables style composition and interpolation across modalities. The work also provides a scalable data-generation pipeline and demonstrates generalization to unseen styles, highlighting practical potential for flexible, multimodal image editing.

Abstract

Given an original image, image editing aims to generate an image that align with the provided instruction. The challenges are to accept multimodal inputs as instructions and a scarcity of high-quality training data, including crucial triplets of source/target image pairs and multimodal (text and image) instructions. In this paper, we focus on image style editing and present StyleBooth, a method that proposes a comprehensive framework for image editing and a feasible strategy for building a high-quality style editing dataset. We integrate encoded textual instruction and image exemplar as a unified condition for diffusion model, enabling the editing of original image following multimodal instructions. Furthermore, by iterative style-destyle tuning and editing and usability filtering, the StyleBooth dataset provides content-consistent stylized/plain image pairs in various categories of styles. To show the flexibility of StyleBooth, we conduct experiments on diverse tasks, such as text-based style editing, exemplar-based style editing and compositional style editing. The results demonstrate that the quality and variety of training data significantly enhance the ability to preserve content and improve the overall quality of generated images in editing tasks. Project page can be found at https://ali-vilab.github.io/stylebooth-page/.

Figures (13)

• Figure 1: Edited images by StyleBooth. Based on multimodal instructions, StyleBooth supports 3 types of image style editing. Following the same instruction template: "Let this image be in the style of <style>/<image>", we conduct (a) text-based style editing and (b) exemplar-based style editing (c) compositional style editing. "<style>" or "<image>" is the identifier of textual style name and visual exemplar images. The style name is placed under the result image and exemplar is shown at the left-bottom corner. In (c), the style name and identifier "<image>" are marked in different color fading levels. The degree of fading represents the proportion of the corresponding style in the result images. Tuned by our elegantly designed style editing data, StyleBooth is capable of generating high-quality output images in diverse styles.
• Figure 2: Overview of StyleBooth method. We propose Multimodal Instruction, mapping the text input and exemplar image input into a same hidden space through a trainable matrix $W$, which unifies vision and text instructions. The textual instruction templates are carefully designed, introducing undetermined identifiers like "<style>" and "<image>" to support multimodal inputs. To balance every style for compositional style editing, we conduct Scale Weights Mechanism $\alpha_i$ on the hidden space embeddings. Editing is conditioned by multimodal features following the composed instructions from different modalities at the same time.
• Figure 3: Iterative Style-Destyle Tuning and Editing pipeline. Following a de-style editing, filtering, style tuning, stylize editing, filtering and de-style tuning steps, Iterative Style-Destyle Tuning and Editing leverages the image quality and usability.
• Figure 4: Generated samples of the intermediate and final image pairs during Iterative Style-Destyle Tuning and Editing. During iterations, image quality gets higher while key style features are gradually wiped off in the de-styled images. We show the style images and de-style results generated in 1st and 2nd de-styled phase and a plain image and results generated in 1st stylize phase.
• Figure 5: Comparisons with instruction-based style editing baselines in Emu Edit benchmark. We show editing results of StyleBooth and 3 baselines. The results of StyleBooth are the most accurate in both style conveying and content preservation comparing to others, though some of the styles and instruction syntax are not contained in our tuning dataset.