StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis

TL;DR

StrokeNUWA introduces stroke tokens as a semantic, compressed representation for vector graphics to leverage LLMs without visual modules. It combines a Vector Quantized-Stroke encoder, an Encoder-Decoder LLM, and an SVG Fixer to produce SVGs from text prompts swiftly, achieving up to 94x faster generation and a 6.9% compression ratio. Across reconstruction and text-guided SVG generation tasks, StrokeNUWA outperforms optimization-based methods and LLM baselines in FID, CLIP, and perceptual quality while delivering semantically richer content. The work highlights the potential of stroke-token representations to enhance semantic fidelity and efficiency in vector-graphic synthesis, with prospects for broader SVG understanding and cross-domain extensions.

Abstract

To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation ''stroke tokens'' on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a 94x speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.

Figures (9)

• Figure 1: Comparison between the visual representation of "grid" token and our proposed "stroke" token. Instead of tokenizing pixels from raster images, we explore a novel visual representation by tokenizing codes, from another image format---Scalable Vector Graphic (SVG). "Stroke" tokens have the following advantages: (1) inherently contain visual semantics, (2) naturally compatible with LLMs, and (3) highly compressed.
• Figure 2: SVG generated by StrokeNUWA. For each image, we provide partial keywords for clarity.
• Figure 3: Overview of VQ-Stroke.
• Figure 4: Architecture of Down-Sample and Up-Sample Blocks.
• Figure 5: Analysis of SVG reconstruction, where $C$ is a constant representing the number of inserted $\texttt{<M>}$ command in PI setting. To facilitate clear observation of the SVG composition, we represent each basic command with a distinct color.