Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation

Peter Schaldenbrand, Zhixuan Liu, Jean Oh

TL;DR

StyleCLIPDraw addresses the problem of generating stylized drawings from text while allowing explicit style control via a sample image. It couples content and style through dual losses on a differentiable, vector-based drawing representation, enabling joint optimization throughout generation. Human evaluations show strong preference for the coupled approach over a decoupled baseline, emphasizing the importance of style in perceived image quality. The work provides open-source code, demonstrations, and a public dataset, highlighting the potential for improved artistic ownership and collaboration with AI, while noting speed and abstract-style challenges as avenues for future work.

Abstract

Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control of the style of image to be generated. We present an approach for generating styled drawings for a given text description where a user can specify a desired drawing style using a sample image. Inspired by a theory in art that style and content are generally inseparable during the creative process, we propose a coupled approach, known here as StyleCLIPDraw, whereby the drawing is generated by optimizing for style and content simultaneously throughout the process as opposed to applying style transfer after creating content in a sequence. Based on human evaluation, the styles of images generated by StyleCLIPDraw are strongly preferred to those by the sequential approach. Although the quality of content generation degrades for certain styles, overall considering both content \textit{and} style, StyleCLIPDraw is found far more preferred, indicating the importance of style, look, and feel of machine generated images to people as well as indicating that style is coupled in the drawing process itself. Our code (https://github.com/pschaldenbrand/StyleCLIPDraw), a demonstration (https://replicate.com/pschaldenbrand/style-clip-draw), and style evaluation data (https://www.kaggle.com/pittsburghskeet/drawings-with-style-evaluation-styleclipdraw) are publicly available.

StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation

TL;DR

StyleCLIPDraw addresses the problem of generating stylized drawings from text while allowing explicit style control via a sample image. It couples content and style through dual losses on a differentiable, vector-based drawing representation, enabling joint optimization throughout generation. Human evaluations show strong preference for the coupled approach over a decoupled baseline, emphasizing the importance of style in perceived image quality. The work provides open-source code, demonstrations, and a public dataset, highlighting the potential for improved artistic ownership and collaboration with AI, while noting speed and abstract-style challenges as avenues for future work.

Abstract

Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control of the style of image to be generated. We present an approach for generating styled drawings for a given text description where a user can specify a desired drawing style using a sample image. Inspired by a theory in art that style and content are generally inseparable during the creative process, we propose a coupled approach, known here as StyleCLIPDraw, whereby the drawing is generated by optimizing for style and content simultaneously throughout the process as opposed to applying style transfer after creating content in a sequence. Based on human evaluation, the styles of images generated by StyleCLIPDraw are strongly preferred to those by the sequential approach. Although the quality of content generation degrades for certain styles, overall considering both content \textit{and} style, StyleCLIPDraw is found far more preferred, indicating the importance of style, look, and feel of machine generated images to people as well as indicating that style is coupled in the drawing process itself. Our code (https://github.com/pschaldenbrand/StyleCLIPDraw), a demonstration (https://replicate.com/pschaldenbrand/style-clip-draw), and style evaluation data (https://www.kaggle.com/pittsburghskeet/drawings-with-style-evaluation-styleclipdraw) are publicly available.
Paper Structure (22 sections, 3 equations, 5 figures, 2 tables)

This paper contains 22 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Synthesis Models
  4. CLIP-Guided Text-to-Image Synthesis
  5. Text-to-Drawing Synthesis
  6. Drawing and Style Evaluation
  7. Method
  8. Drawing Representation
  9. Drawing Augmentation
  10. Content Loss
  11. Style Loss
  12. Optimization Algorithm
  13. Experiments
  14. Decoupled Baseline
  15. Human Evaluation Survey
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Comparison between our (StyleCLIPDraw) method and the baseline (CLIPDraw + Style Transfer). The top row shows the language input followed by the style image input.
  • Figure 2: The top row shows the style image used to generate the following images. The second row used the text prompt "Albert Einstein dancing" and the last row, "A sheep wearing a top hat."
  • Figure 3: Controlling the style of CLIPDraw generated drawings by altering the text description.
  • Figure 4: StyleCLIPDraw optimizes a drawing representation by computing two losses: one for content using the text description and the other for style using a style image. The drawing representation is rasterized, then style and content features are extracted using CLIP and VGG16 models respectively. Style and content features are compared using distance functions to compute a loss.
  • Figure 5: From left to right: increasing the weight of the content loss and decreasing the weight of the style loss ($\lambda_1$ and $\lambda_2$ respectively in Eq. \ref{['eq:objective']}). The drawings were generated using the style image on the far left and the text prompt, "A person is walking down a city street."